I have this query which takes too much time (since last 1 hour is still running) to execute:
select RL.[LINK_ID] as LINK_ID, RPA.[POSTAL_AREA_ID] as POSTAL_AREA_ID, RRN.[STREET_NAME] as STREET_NAME
from RDF_LINK as RL, RDF_POSTAL_AREA as RPA, RDF_ROAD_LINK as RRL, RDF_ROAD_NAME as RRN
where RRL.[ROAD_NAME_ID] = RRN.[ROAD_NAME_ID]
AND RPA.[POSTAL_AREA_ID] IN (RL.[LEFT_POSTAL_AREA_ID], RL.[RIGHT_POSTAL_AREA_ID])
AND RL.[LINK_ID] = RRL.[LINK_ID]
All the columns which are part of the query are indexed.
The ANALYZE command has already been. performed on database.
The database has approx. 73 millions records in the RDF_ROAD_LINK table and same number of records in other tables.
Is there any other way around to write this query?
EXPLAIN QUERY PLAN
select RL.[LINK_ID] as LINK_ID, RPA.[POSTAL_AREA_ID] as POSTAL_AREA_ID, RRN.[STREET_NAME] as STREET_NAME
from RDF_LINK as RL, RDF_POSTAL_AREA as RPA, RDF_ROAD_LINK as RRL, RDF_ROAD_NAME as RRN
where RRL.[ROAD_NAME_ID] = RRN.[ROAD_NAME_ID]
AND RPA.[POSTAL_AREA_ID] IN (RL.[LEFT_POSTAL_AREA_ID], RL.[RIGHT_POSTAL_AREA_ID])
AND RL.[LINK_ID] = RRL.[LINK_ID]
Output ::
0 0 3 SCAN TABLE RDF_ROAD_NAME AS RRN
0 1 2 SEARCH TABLE RDF_ROAD_LINK AS RRL USING INDEX IND_ROAD_NAME_ID (ROAD_NAME_ID=?)
0 2 0 SEARCH TABLE RDF_LINK AS RL USING INDEX sqlite_autoindex_RDF_LINK_1 (LINK_ID=?)
0 3 1 SEARCH TABLE RDF_POSTAL_AREA AS RPA USING COVERING INDEX sqlite_autoindex_RDF_POSTAL_AREA_1 (POSTAL_AREA_ID=?)
0 0 0 EXECUTE LIST SUBQUERY 1
This query returns all 73 million records, and has to look up the corresponding records from the other tables.
This cannot be fast because there is too much data to be cached (and with this size, it's likely that not even the indexes fit into the cache).
In a join between two tables, the database goes through all rows of the first table, and looks up the corresponding row(s) of the second table.
This means that the first table always ends up with a SCAN, because it would not make sense to use an index (going through an index would not be any faster when you need to load all rows anyway).
In this case, using an index for RDF_ROAD_NAME would be possible only if there were an additional filter on an indexed column (WHERE STREET_NAME = 'My Street'), or if the result must be sorted by an indexed column (ORDER BY ROAD_NAME_ID).
If the tables have many columns that are not used in this query, you might be able to speed it up a little bit by using covering indexes (if all data you need is already in the index, the database does not need to look up the corresponding table row):
CREATE INDEX ... ON RDF_ROAD_LINK(ROAD_NAME_ID, LINK_ID);
CREATE INDEX ... ON RDF_LINK(LINK_ID, LEFT_POSTAL_AREA_ID, RIGHT_POSTAL_AREA_ID);
Related
select *
from customers
where column1 = 'test'
limit 5;
I just need 5 records. will execution engine stop running after finding 5 records which matches condition.
I am working on table with millions of records simple select statement with limit is taking ~20 minutes.
Can I improve the performance of this query?
Make sure that you have an index on column1. If not, then the engine has to scan ALL records starting with the first one until it finds 5 matching records. If you know that more than this single column will match your desired rows and also exclude other rows, you could create a compound index consisting of more than one row. You could also consider partitioning your table.
EXPLAIN PLAN FOR
SELECT sightings.sighting_id, spotters.spotter_name,
sightings.sighting_date
FROM sightings
INNER JOIN spotters
ON sightings.spotter_id = spotters.spotter_id
WHERE sightings.spotter_id = 1255;
SELECT plan_table_output
FROM table(dbms_xplan.display('plan_table',null,'basic'));
id Operation Name
0 select statement
1 nested loops
2 table access by index rowid spotters
3 index unique scan pk_spotter_ID
4 table access full sightings
Im trying to understand whats exactly going on here does this sound right:
First the select statement is evaluated and attributes not in the select list are ignored for the output
Nested loop then computes the inner join on spotters.spotters_id = sightings.spotter_id
Table access by index rowid retrieves the rows with the rowids that were returned by step 3 from the spotters table
Index unique scan, scans spotter_id in PK_SPOTTER_ID index and finds rowids associated rows in the spotters table
Table access full, then scans through sightings completely untill sighting_id = 1255 is found
Steps seem to be basically correct but should be buttom-up.
The projection (choosing the relevant columns) is optimally done as early as possible at the scan phase.
The index operation is SEEK (you are not scanning the whole index)
NOTE: THIS ANSWER REFERS TO THE ORIGINAL VERSION OF THE QUESTION.
Oracle is reading the two tables in their entirety.
It is hashing each of the tables based on the join keys -- "re-ordering" the tables so similar keys appear near each other.
It is doing the join.
It is then doing the calculations for the final select and returning the results to the user.
This is what happens, informally, in the right order:
-- The index pk_spotter_id is scanned for at most one row that satisfies spotter_id = 1255
3 index unique scan pk_spotter_ID
-- The spotter_name column is fetched from the table spotters for the previously found row
2 table access by index rowid spotters
-- A nested loop is run for each (i.e. at most one) of the previously found rows
1 nested loops
-- That nested loop will scan the entire sightings table for rows that match the join
-- predicate sightings.spotter_id = spotters.spotter_id
4 table access full sightings
-- That'll be it for your select statement
0 select statement
In general (there are tons of exceptions), Oracle execution plans can be read
Bottom-up
First sibling first
This means that you go down the tree until you find the first leaf operation (e.g. #3), that'll be executed "first", its results are fed to the parent (e.g. #2), all the siblings are then executed top down, all the siblings' results are also fed to the parent, then the parent result is fed to the grand parent (e.g. #1), until you reach the top operation.
This is a very informal explanation of what happens. Do note there will be many exceptions to these rules once statements become more complex.
I've narrowed a performance issue to a particular SQLite query that looks like this:
select *
from test
where (?1 is null or ident = ?1)
and (?2 is null or name = ?2)
and (?3 is null or region = ?3);
This allows any subset of the input parameters (there are more than three) with a single query. Unfortunately, using explain query plan on this yields:
1|0|0|SCAN TABLE test
So SQLite is reading through the entire table no matter what's passed in.
Changing the query to from table indexed by test_idx causes it to fail: Error: no query solution.
Removing the ?1 is null or yields a much more favorable query:
1|0|0|SEARCH TABLE test USING INDEX idx (ident=?)
However, note that only one index can be used. All matches for ident will be scanned looking for matches to other fields. Using a single index that contains all the match fields avoids this:
0|0|0|SEARCH TABLE test USING INDEX test_idx_3 (ident=? AND region=? AND name=?)
It seems reasonable to think that SQLite's query planner would be able to either eliminate or simplify each condition to a simple indexed column check, but apparently that is not the case, as query optimization happens before parameter binding, and no further simplification occurs.
The obvious solution, is to have 2^N separate queries and select the appropriate one at runtime based on which combination of inputs are to be checked. For N=2 or 3 that might be acceptable, but it's absolutely out of the question in this case.
There are, of course, a number of ways to re-organize the database that would make this type of query more reasonable, but assume that's also not practical.
So, how can I search any subset of columns in a table without losing the performance benefit of indexes on those columns?
The only progress I've been able to make is to use a query like this:
select ident, name, region
from test
where (case when ?1 is null then 1 when ident = ?1 then 1 else 0 end)
and (case when ?2 is null then 1 when name = ?2 then 1 else 0 end)
and (case when ?3 is null then 1 when region = ?3 then 1 else 0 end)
This reduces the query to an index scan, rather than a table scan:
0|0|0|SCAN TABLE test USING COVERING INDEX test_idx_3
However, it only works if there's one index containing all the columns of interest, and if the only columns being selected are those in the index. If the index isn't a "covering index" (one containing all the needed values) then SQLite doesn't use the index at all.
The way to get around the second restriction is to structure the query like this:
select ident, name, region, location
from test
where rowid in (
select rowid
from test
where (case when ?1 is null then 1 when ident = ?1 then 1 else 0 end)
and (case when ?2 is null then 1 when name = ?2 then 1 else 0 end)
and (case when ?3 is null then 1 when region = ?3 then 1 else 0 end)
)
yielding:
0|0|0|SEARCH TABLE test USING INTEGER PRIMARY KEY (rowid=?)
0|0|0|EXECUTE LIST SUBQUERY 1
1|0|0|SCAN TABLE test USING COVERING INDEX test_idx_3
This is generally faster than a full table scan, but how much faster depends on several factors:
How much data is in each row that's not in the index? If it's small, then the index scan is almost a table scan.
How many results are there? Each result is a separate primary key search, so for some large number of results in a large table, the N searches will actually be slower than a single pass through the whole table. For M results in a table of N rows, you want O[M log N] << O[N], so m < (N / log N). Call it 3% as a rule of thumb, minus the cost of an index scan:
Don't try to be clever. SQLite's prepared statements do not need much memory, so you actually could keep all 2^N of them. But preparing a query does not need much time, either, so it would be a better idea to construct each query dynamically whenever you need it.
As for the index: the documentation shows that the leftmost columns in the index must be used in the query. This means that you only need a few combinations of columns in your indexes (even for queries that do not use all index columns). In any case, you should prioritize indexes on columns with high selectivity.
I've created an Oracle Text index like the following:
create index my_idx on my_table (text) indextype is ctxsys.context;
And I can then do the following:
select * from my_table where contains(text, '%blah%') > 0;
But lets say we have a have another column in this table, say group_id, and I wanted to do the following query instead:
select * from my_table where contains(text, '%blah%') > 0 and group_id = 43;
With the above index, Oracle will have to search for all items that contain 'blah', and then check all of their group_ids.
Ideally, I'd prefer to only search the items with group_id = 43, so I'd want an index like this:
create index my_idx on my_table (group_id, text) indextype is ctxsys.context;
Kind of like a normal index, so a separate text search can be done for each group_id.
Is there a way to do something like this in Oracle (I'm using 10g if that is important)?
Edit (clarification)
Consider a table with one million rows and the following two columns among others, A and B, both numeric. Lets say there are 500 different values of A and 2000 different values of B, and each row is unique.
Now lets consider select ... where A = x and B = y
An index on A and B separately as far as I can tell do an index search on B, which will return 500 different rows, and then do a join/scan on these rows. In any case, at least 500 rows have to be looked at (aside from the database being lucky and finding the required row early.
Whereas an index on (A,B) is much more effective, it finds the one row in one index search.
Putting separate indexes on group_id and the text I feel only leaves the query generator with two options.
(1) Use the group_id index, and scan all the resulting rows for the text.
(2) Use the text index, and scan all the resulting rows for the group_id.
(3) Use both indexes, and do a join.
Whereas I want:
(4) Use the (group_id, "text") index to find the text index under the particular group_id and scan that text index for the particular row/rows I need. No scanning and checking or joining required, much like when using an index on (A,B).
Oracle Text
1 - You can improve performance by creating the CONTEXT index with FILTER BY:
create index my_idx on my_table(text) indextype is ctxsys.context filter by group_id;
In my tests the filter by definitely improved the performance, but it was still slightly faster to just use a btree index on group_id.
2 - CTXCAT indexes use "sub-indexes", and seem to work similar to a multi-column index. This seems to be the option (4) you're looking for:
begin
ctx_ddl.create_index_set('my_table_index_set');
ctx_ddl.add_index('my_table_index_set', 'group_id');
end;
/
create index my_idx2 on my_table(text) indextype is ctxsys.ctxcat
parameters('index set my_table_index_set');
select * from my_table where catsearch(text, 'blah', 'group_id = 43') > 0
This is likely the fastest approach. Using the above query against 120MB of random text similar to your A and B scenario required only 18 consistent gets. But on the downside, creating the CTXCAT index took almost 11 minutes and used 1.8GB of space.
(Note: Oracle Text seems to work correctly here, but I'm not familiar with Text and I can't gaurentee this isn't an inappropriate use of these indexes like #NullUserException said.)
Multi-column indexes vs. index joins
For the situation you describe in your edit, normally there would not be a significant difference between using an index on (A,B) and joining separate indexes on A and B. I built some tests with data similar to what you described and an index join required only 7 consistent gets versus 2 consistent gets for the multi-column index.
The reason for this is because Oracle retrieves data in blocks. A block is usually 8K, and an index block is already sorted, so you can probably fit the 500 to 2000 values in a few blocks. If you're worried about performance, usually the IO to read and write blocks is the only thing that matters. Whether or not Oracle has to join together a few thousand rows is an inconsequential amount of CPU time.
However, this doesn't apply to Oracle Text indexes. You can join a CONTEXT index with a btree index (a "bitmap and"?), but the performance is poor.
I'd put an index on group_id and see if that's good enough. You don't say how many rows we're talking about or what performance you need.
Remember, the order in which the predicates are handled is not necessarily the order in which you wrote them in the query. Don't try to outsmart the optimizer unless you have a real reason to.
Short version: There's no need to do that. The query optimizer is smart enough to decide what's the best way to select your data. Just create a btree index on group_id, ie:
CREATE INDEX my_group_idx ON my_table (group_id);
Long version: I created a script (testperf.sql) that inserts 136 rows of dummy data.
DESC my_table;
Name Null Type
-------- -------- ---------
ID NOT NULL NUMBER(4)
GROUP_ID NUMBER(4)
TEXT CLOB
There is a btree index on group_id. To ensure the index will actually be used, run this as a dba user:
EXEC DBMS_STATS.GATHER_TABLE_STATS('<YOUR USER HERE>', 'MY_TABLE', cascade=>TRUE);
Here's how many rows each group_id has and the corresponding percentage:
GROUP_ID COUNT PCT
---------------------- ---------------------- ----------------------
1 1 1
2 2 1
3 4 3
4 8 6
5 16 12
6 32 24
7 64 47
8 9 7
Note that the query optimizer will use an index only if it thinks it's a good idea - that is, you are retrieving up to a certain percentage of rows. So, if you ask it for a query plan on:
SELECT * FROM my_table WHERE group_id = 1;
SELECT * FROM my_table WHERE group_id = 7;
You will see that for the first query, it will use the index, whereas for the second query, it will perform a full table scan, since there are too many rows for the index to be effective when group_id = 7.
Now, consider a different condition - WHERE group_id = Y AND text LIKE '%blah%' (since I am not very familiar with ctxsys.context).
SELECT * FROM my_table WHERE group_id = 1 AND text LIKE '%ipsum%';
Looking at the query plan, you will see that it will use the index on group_id. Note that the order of your conditions is not important:
SELECT * FROM my_table WHERE text LIKE '%ipsum%' AND group_id = 1;
Generates the same query plan. And if you try to run the same query on group_id = 7, you will see that it goes back to the full table scan:
SELECT * FROM my_table WHERE group_id = 7 AND text LIKE '%ipsum%';
Note that stats are gathered automatically by Oracle every day (it's scheduled to run every night and on weekends), to continually improve the effectiveness of the query optimizer. In short, Oracle does its best to optimize the optimizer, so you don't have to.
I do not have an Oracle instance at hand to test, and have not used the full-text indexing in Oracle, but I have generally had good performance with inline views, which might be an alternative to the sort of index you had in mind. Is the following syntax legit when contains() is involved?
This inline view gets you the PK values of the rows in group 43:
(
select T.pkcol
from T
where group = 43
)
If group has a normal index, and doesn't have low cardinality, fetching this set should be quick. Then you would inner join that set with T again:
select * from T
inner join
(
select T.pkcol
from T
where group = 43
) as MyGroup
on T.pkcol = MyGroup.pkcol
where contains(text, '%blah%') > 0
Hopefully the optimizer would be able to use the PK index to optimize the join and then appy the contains predicate only to the group 43 rows.
I'm reading about indexes in my database book and I was wondering if I was correct in my assumption that a WHERE clause with a non-constant expression in it will not use the index.
So if i have
SELECT * FROM statuses WHERE app_user_id % 10 = 0;
This would not use an index created on app_user_id. But
SELECT * FROM statuses WHERE app_user_id = 5;
would use the index on app_user_id.
Usually (there are other options) a database index is a B-Tree, which means that you can do range scans on it (including equality scans).
The condition app_user_id % 10 = 0 cannot be evaluated with a single range scan, which is why a database will probably not use an index.
It could still decide to use the index in another way, namely for a full scan: Reading the whole table takes more time than just reading the whole index. On the other hand, after reading the index you may still get back to the table, so the overall cost may end up being higher.
This is up to the database query optimizer to decide.
A few examples:
select app_user_id from t where app_user_id % 10 = 0
Here, you do not need the table at all, all necessary data is in the index. The database will most likely do a full index scan.
select count(*) from t where app_user_id % 10 = 0
Same. Full index scan.
select count(*) from t
Only if app_user_id is NOT NULL can this be done with the index (because NULL data is not in the index, at least on Oracle, at least on single column indexes, your database may handle this differently).
Some databases do not need to do access table or index for this, they maintain row counts in the metadata.
select * from t where app_user_id = 5
This is the classic scenario for an index. The database can look at the small section of the index tree, retrieve a small (just one if this was a unique or primary index) number of rowids and fetch those selectively from the table.
select * from t where app_user_id between 5 and 10
Another classic index case. Range scan in the tree returns a small number of rowids to fetch from the table.
select * from t where app_user_id between 5 and 10 order by app_user_id
Since index scans return ordered data, you even get the sorting for free.
select * from t where app_user_id between 5 and 1000000000
Maybe here you should not be using an index. It seems to match too many records. This is a case where having bind variables hide the range from the database could actually be detrimental.
select * from t where app_user_id between 5 and 1000000000
order by app_user_id
But here, since sorting would be very expensive (even taking up temporary swap disk space), maybe iterating in index order is good. Maybe.
select * from t where app_user_id % 10 = 0
This is difficult to decide. We need all columns, so ultimately the query needs to touch the table. The question is whether to go through an index first. The query returns approximately 10% of the whole table. That is probably too much for an index access path to be efficient. If the optimizer has reason to believe that the query returns much less than 10% of the table, an index scan followed by accessing the table might be good. Same if the table is very fragmented (lots of deleted rows eating up space).