I've found a number of resources that talk about tuning the database server, but I haven't found much on the tuning of the individual queries.
For instance, in Oracle, I might try adding hints to ignore indexes or to use sort-merge vs. correlated joins, but I can't find much on tuning Postgres other than using explicit joins and recommendations when bulk loading tables.
Do any such guides exist so I can focus on tuning the most run and/or underperforming queries, hopefully without adversely affecting the currently well-performing queries?
I'd even be happy to find something that compared how certain types of queries performed relative to other databases, so I had a better clue of what sort of things to avoid.
update:
I should've mentioned, I took all of the Oracle DBA classes along with their data modeling and SQL tuning classes back in the 8i days ... so I know about 'EXPLAIN', but that's more to tell you what's going wrong with the query, not necessarily how to make it better. (eg, are 'while var=1 or var=2' and 'while var in (1,2)' considered the same when generating an execution plan? What if I'm doing it with 10 permutations? When are multi-column indexes used? Are there ways to get the planner to optimize for fastest start vs. fastest finish? What sort of 'gotchas' might I run into when moving from mySQL, Oracle or some other RDBMS?)
I could write any complex query dozens if not hundreds of ways, and I'm hoping to not have to try them all and find which one works best through trial and error. I've already found that 'SELECT count(*)' won't use an index, but 'SELECT count(primary_key)' will ... maybe a 'PostgreSQL for experienced SQL users' sort of document that explained sorts of queries to avoid, and how best to re-write them, or how to get the planner to handle them better.
update 2:
I found a Comparison of different SQL Implementations which covers PostgreSQL, DB2, MS-SQL, mySQL, Oracle and Informix, and explains if, how, and gotchas on things you might try to do, and his references section linked to Oracle / SQL Server / DB2 / Mckoi /MySQL Database Equivalents (which is what its title suggests) and to the wikibook SQL Dialects Reference which covers whatever people contribute (includes some DB2, SQLite, mySQL, PostgreSQL, Firebird, Vituoso, Oracle, MS-SQL, Ingres, and Linter).
As for badly performing queries - do explain analyze and read it.
You can put explain analyze output on site like explain.depesz.com - it will help you find the elements that really take the most time.
There is a nice online tool that takes the output of EXPLAIN ANALYZE, and graphically shows you critical parts (e.g. wrong estimates, hot spots, etc)
http://explain.depesz.com/help
Btw, I think posted queries become public, and the "previous explains" link has been hit by spambots.
http://www.postgresql.org/docs/current/static/indexes-examine.html
You can give hints: SET enable_indexscan TO false; would make PostgreSQL try to not use indexes
To address your point, unfortunately the only way to tune a query in Postgres is pretty much to tune the database underlying it. In oracle, you can set all of those options on a query by query basis, trump the optimizers plan in the process, but in Postgres, you're pretty much at the mercy of the optimizer, for good and ill.
The PGAdmin3 tool includes a graphical explanation tool for breaking down how a query is handled. It also is especially helpful for showing where table scans occur.
Best I've seen are in here: http://wiki.postgresql.org/wiki/Using_EXPLAIN, but the latest PDF in there is from 2008, so there may be something more recent. I'm interested to hear other user's answers.
Also, something's brewing in the contrib packages: http://www.sai.msu.su/~megera/wiki/plantuner
Related
I am doing a paper about query optimization in different DBMS, and I am trying to find core differences in those.
Both use CBO, cost based optimization in the same way, parse the query -> generate plans -> pick best one given statistics about the database.
I'm still researching information on those two engines, but if someone knows how they differ (or not) will be appreciated.
Not a comprehensive answer at all, but wanted to give you my insight. In short, Oracle has a much more developed SQL optimizer.
For starters, Oracle has much more algorithms to choose from. This means, sometimes Oracle distinguish between subtle differences and offer, let's say, three algorithms; MySQL (under the same circumstances) only has one to choose from. Therefore, Oracle has better options for particular cases.
Another difference is that MySQL's execution plans are not very readable. I'm not saying they are bad internally, just that the explain extended doesn't tell you many specifics. Oracle makes a very clear difference between access and filter predicates, while in MySQL you don't really know what's going on.
Oracle has many algorithms suitable for parallel processing in multi servers, while MySQL is limited to multiple thread in the same machine. This can make a difference for highly parallelisable queries than benefit for multi-servers.
Oracle still has a RBO (Rule-Based Optimizer) than can be useful on some occasions. MySQL doesn't. Anyway, Oracle recommends not to use it, but it's still there if you need it.
Oracle offers a myriad of "hints" to the optimizer in the form of comments (/* ... */ as far as I remember?) where you can tweak the execution plan to suit your needs. MySQL has fewer "clauses" for this.
This question already has an answer here:
When to use Oracle hints?
(1 answer)
Closed 5 years ago.
I have gone through some documentation on the net and using hints is mostly discouraged. I still have doubts about this. Can hints be really useful in production specially when same query is used by hundreds of different customer.
Is hint only useful when we know the number of records that are present in the tables? I am using leading in my query and it gives faster results when the data is very large but the performance is not that great when the records fetched are less.
This answer by David is very good but I would appreciate if someone clarified this in more details.
Most hints are a way of communicating our intent to the optimizer. For instance, the leading hint you mention means join tables in this order. Why is this necessary? Often it's because the optimal join order is not obvious, because the query is badly written or the database statistics are inaccurate.
So one use of hints such as leading is to figure out the best execution path, then to figure out why the database doesn't choose that plan without the hint. Does gathering fresh statistics solve the problem? Does rewriting the FROM clause solve the problem? If so, we can remove the hints and deploy the naked SQL.
Some times there are times where we cannot resolve this conundrum, and have to keep the hints in Production. However this should be a rare exception. Oracle have had lots of very clever people working on the Cost-Based Optimizer for many years, so its decisions are usually better than ours.
But there are other hints we would not blink to see in Production. append is often crucial for tuning bulk inserts. driving_site can be vital in tuning distributed queries.
Conversely other hints are almost always abused. Yes parallel, I'm talking about you. Blindly putting /*+ parallel (t23, 16) */ will probably not make your query run sixteen times faster, and not infrequently will result in slower retrieval than a single-threaded execution.
So, in short, there is no universally applicable advice to when we should use hints. The key things are:
understand how the database works, and especially how the cost-based optimizer works;
understand what each hint does;
test hinted queries in a proper tuning environment with Production-equivalent data.
Obviously the best place to start is the Oracle documentation. However, if you feel like spending some money, Jonathan Lewis's book on the Cost-Based Optimizer is the best investment you could make.
I couldn't just rephrase that, so I will paste it here
(it's a brief explanation as of "When Not To Use Hints", that I had bookmarked):
In summary, don’t use hints when
What the hint does is poorly understood, which is of course not limited to the (ab)use of hints;
You have not looked at the root cause of bad SQL code and thus not yet tapped into the vast expertise and experience of your DBA in tuning the database;
Your statistics are out of date, and you can refresh the statistics more frequently or even fix the statistics to a representative state;
You do not intend to check the correctness of the hints in your statements on a regular basis, which means that, when statistics change, the hint may be woefully inadequate;
You have no intention of documenting the use of hints anyway.
Source link here.
I can summarize this as: The use of hints is not only a last resort, but also a lack of knowledge on the root cause of the issue. The CBO (Cost Based Optimizer) does an excellent job, if you just ensure some basics for it. Those include:
Fresh statistics
1.1. Index statistics
1.2. Table statistics
1.3. Histograms
Correct JOIN conditions and INDEX utilization
Correct Database settings
This article here is worth reading:
Top 10 Reasons for poor Oracle performance
Presented by non other, but Mr. Donald Burleson.
Cheers
In general hints should be used only exceptional, I know following situations where they make sense:
Workaround for Oracle bugs
Example: Once for a SELECT statement I got an error ORA-01795: maximum expression number in list - 1000, although the query did not contain an IN expression at all.
Problem was: The queried table contains more than 1000 (sub-) partitions and Oracle did a transformation of my query. Using the (undocumented) hint NO_EXPAND_TABLE solved the issue.
Datewarehouse application while staging
While staging you can have significant changes on your data where the table/index statistics are not aware about as statistics are gathered only once a week by default. If you know your data structure then hints could be useful as they are faster than running DBMS_STATS.GATHER_TABLE_STATS(...) manually all the time in between your operations. On the other hand you can run DBMS_STATS.GATHER_TABLE_STATS() even for single columns which might be the better approach.
Online Application Upgrade Hints
From Oracle documentation:
The CHANGE_DUPKEY_ERROR_INDEX, IGNORE_ROW_ON_DUPKEY_INDEX, and
RETRY_ON_ROW_CHANGE hints are unlike other hints in that they have a
semantic effect. The general philosophy explained in "Hints" does not
apply for these three hints.
Does oracle allow creation of mnemonics for hints in SQL? I am not able to find any such concept in Oracle docs but I have found certain queries running in production with hints like the below one:
select /*+ AHintWhichIsNotInOracleDocumentation */ from some_table;
I thought optimizer would safely ignore this until I found http://www.confio.com/logicalread/oracle-11g-making-query-run-magically-faster-mc02/#.U_D8WqOTJUA
The author talks about "adding" a hint called "RICHS_SECRET_HINT" in the "X$" tables? Is this feature available in Oracle? If yes, links to docs please. Thanks.
Edit:
To clarify further, I am looking to find if there Oracle provides ways to create a key-value sort of relationship between new hints and Oracle provided hints. This seems to be a pointless feature considering it is good only to shorten the length of the hints when there are a lot of hints used in a SQL which is rarely the case. But considering the hints I saw in work, I am more curious to find if they exists or not.
So in essence in the above SQL, I am expecting some mapping between AHintWhichIsNotInOracleDocumentation and Oracle's standard hints like ORDERED, USE_NL, etc.,
The article you've linked to has nothing to do with creating your own hint. The article is demonstrating that it is relatively easy to trick yourself into thinking that you have improved the performance of a query (in this case by adding a hint that does nothing) when the reality is that the performance improved only because data has been cached by the prior executions.
You cannot define your own hints. There are hints that are undocumented. Given how rarely using a documented hint is the proper long-term answer to improving query performance (it almost always makes sense to fix the underlying statistics issue or to create a profile/ outline/ etc.), it would be exceedingly unlikely that you'd want to use an undocumented hint. I can't imagine a case where it would make sense to be able to define your own hint.
I was just wondering where I could find more information on these optimizations? Google searches tend to emphasize prepared queries and such, and not really at the angle of the abstraction the SQL provides.
Source:
http://www.joelonsoftware.com/articles/LeakyAbstractions.html
The SQL language is meant to abstract
away the procedural steps that are
needed to query a database, instead
allowing you to define merely what you
want and let the database figure out
the procedural steps to query it. But
in some cases, certain SQL queries are
thousands of times slower than other
logically equivalent queries. A famous
example of this is that some SQL
servers are dramatically faster if you
specify "where a=b and b=c and a=c"
than if you only specify "where a=b
and b=c" even though the result set is
the same. You're not supposed to have
to care about the procedure, only the
specification. But sometimes the
abstraction leaks and causes horrible
performance and you have to break out
the query plan analyzer and study what
it did wrong, and figure out how to
make your query run faster.
Looking at MySql in particular.
You can try SQL Server Performance, although I think it's geared towards MS SQL Server more than other RDBMSs. Personally, I look at performance tuning as a process more than a collection of tidbits.
Once you get the process down you're likely to come across single item optimizations as you go, but it's the process itself that will give you the most bang for your buck. Learn how to read query plans (or the equivalent in your RDBMS), learn the insides/behind the scenes implementation of your server, how it stores and uses indexes, how to find bottlenecks in IO, memory, locking, etc.
Books are better than web searches to learn performance tuning for a database. It is a complex subject and varies greatly from datbase to database and even as #OMGPonies said from version to version.
Only My SQL Performance book I found at amazon, don;t know how good it is:
http://www.amazon.com/High-Performance-MySQL-Optimization-Replication/dp/0596101716/ref=sr_1_1?ie=UTF8&s=books&qid=1277756707&sr=8-1
"these" are not optimizations.
Learn profiling - the source of all optimizations.
That's all you need.
One you mentioned is not "optimization tidbit". It was an example of totally different subject.
And it is not supposed for blind usage.
But only as a result of profiling, if applicable.
The whole your approach is wrong. There are no "optimization tidbits". There are only profiling. Once you find your "where a=b and b=c" query runs slow - you can start looking for the solution, not sooner.
So, you have 2 instruments to use:
BENCHMARK your query goes here
and
EXPLAIN your query goes here
study their output and then ask particular questions, regarding your server, your settings, your database. That's the only way. No "canned recipe" could help.
As for just a curious reading, you can follow blog, named surprisingly http://mysqlperformanceblog.com
Anyone have any idea? And any open source sofware which also seens to perform this kind of functionality?
I am not sure what you are asking, it is pretty straightforward, you type in your SQL and SQLLab Xpert tries many combinations of rewriting your query and runs them all, selecting the fastest. I find the approach a little dubious, you probably will get something that runs faster than what you originally had, but probably not the fastest possible (unless it is very simple SQL).
I prefer to hand tune, the Oracle performance manual http://download.oracle.com/docs/cd/B28359_01/server.111/b28274/toc.htm, Chapters 11-20, has all the information you need, in my opinion better than the shotgun approach SQLLab Xpert takes.