My DBA thinks the new Oracle12c optimizer features means he doesn't need to add indexes on important columns anymore. I am having a hard time believing that can possibly be true. It only seems to improve how joins are performed based on historical statistics.
You are right, he has totally misunderstood. You can read about Adaptive Query Optimization in this Oracle white paper. It means that the optimizer can change it query execution plan while it is running. But if you don't index columns that could benefit from them, the optimizer will not be able to (adaptively or otherwise) choose a plan that would benefit from those indexes!
Related
Is is possible to somehow optimize the performance of the queries (apart from playing with hardware and OS settings) under these conditions
1) You can't add indexes.
2) You can't alter the queries themselves.
This is the common constraint while bench-marking the performance of a database.
I understand that the dbms has a query optimizer that plays a number game with all the statistics pertaining to accessing the tables touched by the query. Are there cases when the query optimizer comes up with sub optimal solutions. I know that you can force the optimizer to use a particular query plan. Not sure how to cache it though without altering the query plan. DB in question is Sybase
Independent of the specific case here (Sybase), there are multiple ways to optimize a query under the given conditions. Syntax is system-specific.
Most systems rely on statistics for finding the best query plan. So updating the statistics could help improve performance.
Many systems allow to set an optimization level independent of the application. This can have positive impact on the performance.
Many systems allow to re-use query plans for similar ad-hoc queries (dynamic SQL). Usually this has positive impact.
Allowing the database system (independent to the OS) to assign more memory to bottlenecks can also help.
What privileges do you have, what are the benchmark rules?
Data Henrik mentions optimisation level - you can set this system-wide for Sybase, or per session.
You can even have a flexible method that sets the level according to application name or login Id (see Rob Verschoor's Sybase site - login triggers.) I'd guess if you're not allowed to change queries or indexes you'd not likely be allowed to do this.
As far as I can tell you don't have a specific problem - you just mention benchmarking.
You should be sure all tables have UPDATE INDEX STATISTICS run on them, and you could then do your benchmarks with the 3 Sybase optimisation levels - OLTP, MIX, DSS.
If you have specific problems, that's another subject.
Is there a way to find out which queries benefit from a particular index?
I have used the DMV views and I know the index is being used in production but it would be great if there was a way to get a list of the queries positively impacted so I can make a decision if each index is worth keeping.
EDIT: I am using SQL Server
Thanks for your help!
Speaking from Oracle point of view: in Oracle, I can inspect query plan which gives enough information to guess whether an index was used or not. Remember that optimizer makes decisions based on the SQL at hand. There is no hard and fast rule re permanent use or non-use of an index. So, even if you find out that an index is being used or not, you can [almost] always modify the query so that the opposite is true!
Speaking of positive impact: again, it will only tell you how things are at this moment. For example, a table doesn't have enough records and a full table scan may be faster than using an index (due to overhead involved). But what if the situation changes (e.g. lot more records are inputted into that table)?
Bottom line: hard to make these decisions on permanent basis just by looking at what optimizer decided today, or what statistics are maintained by DB at this moment. Your knowledge of the data, its design and structure, and how it is being queried will be the real key on making these decisions.
My guess is that you are asking this question because you have lots of indices and you would like to get rid of a few. Unless the data changes rapidly, there is little overhead in maintaining indices (storage is cheap!). If that is the case, let's hope that optimizer is smart enough to make decision about using or not using an index based on cost... :-)
There have been several questions recently about database indexing and clustered indexing and it has been kind of new to me until the last couple weeks. I was wondering how important it is and what kind of performance gains can be expected from creating them.
Edit: What is usually the best type of fields to look at when putting in a clustered index when you are first starting out?
Very veryA(G,G) important. In my opinion, wise indexing is the absolute most important thing in DB performance optimization.
This is not an easy topic to cover in a single answer. Good indexing requires knowledge of queries going to happen on the database, making a large number of trade-offs and understanding the implication of a specific index in the specific DB engine. But it's very important nevertheless.
EDIT: Basically, clustered indexes usually should have short lengths. They should be created on queries which reflect a range. They should not have duplicate entries. But these guidelines are very general and by no means the right thing. The right thing is to analyze the queries that are gonna be executed. Carefully benchmarking and analyzing execution plans and understanding what is the best way to do it. This requires years of experience and knowledge and by no means it's something to explain in a single paragraph. It's the primary thing that makes DB experts expert (It's not the only thing, but it's primitive to other important things, such as concurrency issues, availability, ...)!
Indexing: extremely important. Having the wrong indexes makes queries harder, sometimes to the point they can't be completed in a sensible time.
Indexes also impact insert performance and disc usage (negatively), so keeping lots of superfluous indexes around on large tables is a bad idea too.
Clustering is something worth thinking about, I think it's really dependent on the behaviour of the specific database. If you can cluster your data correctly, you can dramatically reduce the amount of IOPs required to satisfy requests for rows not in memory.
Without proper indexes, you force the RDBMS to do table scans to query for anything. Terribly inefficient.
I'd also infer that you don't have primary keys, which is a cardinal sin in relational design.
Indexing is very important when the table contains many rows.
With a few rws, performance is better without indexes.
With larger tables indexes are very important to get good performance.
It is not easy to defined them. Clustered means that the data are stored in the clustered index order.
To get good hints of indexes you could use Toad
Indexing is vitally important.
The right index for a query can improve performance so dramatically it can seem like witchcraft.
As the other answers have said, indexing is crucial.
As you might infer from other answers, clustered indexing is much less crucial.
Decent indexing gives you first order performance gains - orders of magnitude are common.
Clustered indexing is a second order or incremental performance gain - usually giving small (<100%) percentages of performance increase.
(We also get into questions of 'what is a 100% performance gain'; I'm interpreting the percentage as ((oldtime - newtime)/newtime) * 100, so if the old time is 10 seconds and the new time is 5 seconds, the performance increase is 100%.)
Different DBMS have different interpretations of what a clustered index means. Beware.
In particular, some DBMS cluster the data once and thereafter, the clustering decays over time until the data is reclustered. Others take a more active view of clustering, I believe.
The clustered index is ususally but not always your primary key. One way of looking at a clustered index is to think of the data being physically ordered based on the values of the clustered index.
This may very well not be the case in reality however refrencing clustered indexes ususally gets you the following performance bonuses anyway:
All columns of the table are accessable for free when resolved from a clustered index hit as if they were contained within a covering index. (A query resolvable using just the index data without having to refrence the data pages of the table itself)
Update operations can be made directly against a clustered index without intermediate processing. If you are doing a lot of updates against a table you ususally want to be refrencing the clustered columns.
Depending on implementation there may be a sequential access benefit where data stored on disk is retreived quicker with fewer expensive disk seek operations.
Depending on implementation there may be free index benefit where a physical index is not necessary as data access can be resolved via simple guessing game algorithms.
Don't count on #3 and especially #4. #1 and #2 are ususally safe bets on most RDBMS platforms.
I am an advocate of ORM-solutions and from time to time I am giving a workshop about Hibernate.
When talking about framework-generated SQL, people usually start talking about how they need to be able to use "hints", and this is supposedly not possible with ORM frameworks.
Usually something like: "We tried Hibernate. It looked promising in the beginning, but when we let it loose on our very very complex production database it broke down because we were not able to apply hints!".
But when asked for a concrete example, the memory of those people is suddenly not so clear any more ...
I usually feel intimidated, because the whole "hints"-topic sounds like voodoo to me...
So can anybody enlighten me? What is meant by SQL-hints or DB-Hints?
The only thing I know, that is somehow "hint-like" is SELECT ... FOR UPDATE. But this is supported by the Hibernate-API...
A SQL statement, especially a complex one, can actually be executed by the DB engine in any number of different ways (which table in the join to read first, which index to use based on many different parameters, etc).
An experienced dba can use hints to encourage the DB engine to choose a particular method when it generates its execution plan. You would only normally need to do this after extensive testing and analysis of the specific queries (because the DB engines are usually pretty darn good at figuring out the optimum execution plan).
Some MSSQL-specific discussion and syntax here:
http://msdn.microsoft.com/en-us/library/ms181714.aspx
Edit: some additional examples at http://geeks.netindonesia.net/blogs/kasim.wirama/archive/2007/12/31/sql-server-2005-query-hints.aspx
Query hints are used to guide the query optimiser when it doesn't produce sensible query plans by default. First, a small background in query optimisers:
Database programming is different from pretty much all other software development because it has a mechanical component. Disk seeks and rotational latency (waiting fora particular sector to arrive under the disk head) are very expensive in comparison to CPU. Different query resolution strategies will result in different amounts of I/O, often radically different amounts. Getting this right or wrong can make a major difference to the performance of the query. For an overview of query optimisation, see This paper.
SQL is declarative - you specify the logic of the query and let the DBMS figure out how to resolve it. A modern cost-based query optimiser (some systems, such as Oracle also have a legacy query optimiser retained for backward compatibility) will run a series of transformations on the query. These maintain semantic equivalence but differ in the order and choice of operations. Based on statistics collected on the tables (sizes, distribution histograms of keys) the optimiser computes an estimate of the amount of work needed for each query plan. It selects the most efficient plan.
Cost-based optimisation is heuristic, and is dependent on accurate statistics. As query complexity goes up the heuristics can produce incorrect plans, which can potentially be wildly inefficient.
Query hints can be used in this situation to force certain strategies in the query plan, such as a type of join. For example, on a query that usually returns very small result sets you may wish to force a nested loops join. You may also wish to force a certain join order of tables.
O/R mappers (or any tool that generates SQL) generates its own query, which will typically not have hinting information. In the case that this query runs inefficiently you have limited options, some of which are:
Examine the indexing on the tables. Possibly you can add an index. Some systems (recent versions of Oracle for example) allow you index joins across more than one table.
Some database management systems (again, Oracle comes to mind) allow you to manually associate a query plan with a specific query string. Query plans are cached by a hash value of the query. If the queries are paramaterised the base query string is constant and will resolve to the same hash value.
As a last resort, you can modify the database schema, but this is only possible if you control the application.
If you control the SQL you can hint queries. In practice it's fairly uncommon to actually need to do this. A more common failure mode on O/R mappers with complex database schemas is they can make it difficult to express complex query predicates or do complex operations over large bodies of data.
I tend to advocate using the O/R mapper for the 98% of work that it's suited for and dropping to stored procedures where they are the appropriate solution. If you really need to hint a query than this might be the appropriate strategy. Unless there is something unusual about your application (for example some sort of DSS) you should
only need to escape from the O/R mapper on a minority of situations. You might also
find (again, an example would be DSS tools working with the data in aggregate) that an O/R mapper is not really the appropriate strategy for the application.
While HINTS do as the other answers describe, you should only use them in rare, researched circumstances. 9 times out of 10 a HINT will result in a poor query plan. Unless you really know what you are doing, don't use them.
There is no such thing as "optimized SQL code", because SQL code is never executed.
SQL code is translated into an execution plan by the Optimizer. The Optimizer will use the information it has to choose (among other things).
the order in which tables are involved
the join method for each involved table (nested/merge/hash)
how to access a table's data (direct table access/ index with bookmark lookup/direct index access) (scan/seek)
should parallelism be used, and when to end parallelism (gather streams)
Query hints allow a programmer to over-ride (in most cases) or suggest politely (in other cases) the optimizer's choices.
Query hints can let you force off parallelism, force all joins to be implemented as nested loop, force one index to be used over another... as a few examples.
Since the optimizer is really good, if one over-rides the optimizer, one is generally asking for a non-optimal plan. Query hints are best served when the optimizer does not have the required information to make a good choice.
One place I use query hints is for table variables. Table variables are assumed to have 0 rows by the Optimizer, and so the Optimizer always joins table variables using nested loop (the best join implementation for small numbers of rows). If I have a large table variable - already ordered in a favorable way for merge join, I can specify a merge join be used by applying a query hint.
All modern RDBMS-es have some sort of query optimizer that calculates best query plan, which is sequence of read/write operations needed to execute SQL query.
Sometimes plans can be suboptimal, so RDBMS designers included "hints" in SQL. Hints are instructions you can embed in your SQL that affect query optimizer, With hints you can instruct query optimizer e.g. which indexes it should use, in what order data should be read from tables, ...
So, with hints you can resolve some bottlenecks that the query optimizer cannot solve by itself.
For example, here is list of Oracle hints.
Oracle decided to dismiss the rule-based optimizer from version 10g, leaving the cost-based one as the only choice.
I think that a rule-based optimizer has the unvaluable positive side of being always predictable. I've seen instead Oracle 10g changing execution plans from night to day, leading to turtle-like performances.
Which could be the rationale behind this change?
Because everything you can do with RBO, can be done with CBO.
The CBO can be rule-based too — more than that, you may decide the "rules" yourself.
To create your own "rules", you hint your query or do a CREATE OUTLINE which will hint it for you. As a result, your execution plan is stable.
The outlines are stored in a system schema called OUTLN, they are editable.
As for me, I always supply hints to my queries running in a production database.
The RBO is often predicatably bad as well as predictably good. It also doesn't support partitioning and some other database features. The CBO is much better, and as Quassnoi says plan stability is a feature of the CBO also.
The RBO has been deprecated for a long time; it was really just retained for backwards compatibility with legacy applications. Oracle have been announcing the demise of the RBO since (IIRC) version 8, which came out about 10 years ago.
The RBO was deterministic, but not all that clever. Oracle was originally designed before cost-based optimisers were even available, let alone a mature technology. The RBO has been frozen for a long time and does not support a lot of features of modern Oracle engines.
Cost-based optimisation is much smarter. However, if you had queries optimised for the RBO, they might not play nicely with the CBO. You will probably have to re-write or hint your queries appropriately to tune them for the CBO. There is also a facility to specify a query plan and override the CBO with that plan. This will give you deterministic query execution with stable plans.
(I am not a DBA.)
My understanding is that Oracle has been moving away from the RBO for a long time in favor of CBO. It seems useful to me to stop supporting a feature that is no longer in active development (given a long enough depreciation period) so that everyone is using the most effective features.
It's interesting that you called predictability an "unvaluable" effect of using the rule-based optimizer. It seems like when the data changes to make an execution plan sub-optimal it would be best to switch to a new one. Only in the case you alluded to where the optimizer flip-flops between two execution plan would there be a problem with picking the best plan for the the data you are actually querying. I'm not sure what advantage predictability is in more normal situation.
Ending support of the out-dated optimizer ought to free up support for the newer optimizer.
The reason they moved to cost-based optimization is that it can perform better since its based on analyzing statistical information that the rule-based optimizer does not have.
To make the CBO work better, its important to understand the role that statistics gathering plays in execution plan changes which directly affect performance. For one thing, running statistics more or less frequently could help you. Here is a good article about the CBO and statistics:
Optimizing Oracle Optimizer Statistics
I think you should do rule based programming. Don't think about the situation, follow a list of inviolate rules, no matter what the situation, no matter what you think is the better way, if the rules say use a FOR LOOP in case X then you have to use a loop, even if you know if there will only be 1, loop from 1 to 1.
Stipulate:
Every query has a best plan.
Every query optimizer will determine that plan x% of the time.
The RBO had nowhere else to go, it's percent accuracy is lower than the CBO to be sure, but it was never going to get any better. It was limited like any rule based system.