As per this article,
First Rule: You can’t use a function in the left side (the field side) of the predicate.
I have a table with a field TOTAL_ARREARS declared as
[TOTAL_ARREARS] [nvarchar](50) NULL
and I cannot change that.
Now the query is a simple one
select Id,name
from tbl
where CAST(TOTAL_ARREARS AS FLOAT)> 0
The execution plan shows
How can I apply the First Rule of the article here (avoiding function on the left)?
Edit
I tried with WHERE TOTAL_ARREARS NOT IN('0.000','.000') that though solved but not happy with the solution.
Is there anything better?
First of the table design should be looked into where we are using a varchar column to store numeric values.
In case of Oracle database you can always explore the option of function based index to get around of these kind of issues. In case of function based indexes the index key will be based on functions rather than the actual column value. It will only be used when you use the same exactly the same string(including case of letters) in the predicate.
In case of databases where function based index is not available you can consider creating an additional column as mentioned in the following post for sql server.
Function-based indexes in SQL Server
Related
I built sqlite3 from source to include the FTS3 support and then created a new table in an existing sqlite database containing 1.5million rows of data, using
CREATE VIRTUAL TABLE data USING FTS3(codes text);
Then used
INSERT INTO data(codes) SELECT originalcodes FROM original_data;
Then queried each table with
SELECT * FROM original_data WHERE originalcodes='RH12';
This comes back instantly as I have an index on that column
The query on the FTS3 table
SELECT * FROM data WHERE codes='RH12';
Takes almost 28 seconds
Can someone help explain what I have done wrong as I expected this to be significantly quicker
The documentation explains:
FTS tables can be queried efficiently using SELECT statements of two different forms:
Query by rowid. If the WHERE clause of the SELECT statement contains a sub-clause of the form "rowid = ?", where ? is an SQL expression, FTS is able to retrieve the requested row directly using the equivalent of an SQLite INTEGER PRIMARY KEY index.
Full-text query. If the WHERE clause of the SELECT statement contains a sub-clause of the form " MATCH ?", FTS is able to use the built-in full-text index to restrict the search to those documents that match the full-text query string specified as the right-hand operand of the MATCH clause.
If neither of these two query strategies can be used, all queries on FTS tables are implemented using a linear scan of the entire table.
For an efficient query, you should use
SELECT * FROM data WHERE codes MATCH 'RH12'
but this will find all records that contain the search string.
To do 'normal' queries efficiently, you have to keep a copy of the data in a normal table.
(If you want to save space, you can use a contentless or external content table.)
You should read documentation more carefully.
Any query against virtual FTS table using WHERE col = 'value' will be slow (except for query against ROWID), but query using WHERE col MATCH 'value' will be using FTS and fast.
I'm not an expert on this, but here are a few things to think about.
Your test is flawed (I think). You are contrasting a scenario where you have an exact text match (the index can be used on original_data - nothing is going to outperform this scenario) with an equality on the fts3 table (I'm not sure that FTS3 would even come into play in this type of query). If you want to compare apples to apples (to see the benefit of FTS3), you're going to want to compare a "like" operation on original_data against the FTS3 "match" operation on data.
I am creating sql queries and I have read in one book that when using NOT operator or LIKE operators Indexes do not work. How much true this statement is. How can we avoid this if the statement is true. How a query should be made to do the same work avoiding these operators.
What all other areas are there in sql server where Indexes are deferred.
Like statements that wildcard the leftside can not use an index if one is defined for the column:
WHERE column LIKE '%abc'
WHERE column LIKE '%abc%'
But either of these can use an index:
WHERE column LIKE 'abc%'
WHERE column LIKE 'abc'
Frankly, use LIKE for very simple text searching - if you're really needing a text search that performs well, look at Full Text Searching (FTS). Full Text Searching has it's own indexes.
The decision really comes down to the optimizer for which index to use, but something that ensures that an index will not be used it to wrap column values in function calls. This for example will not use indexes:
WHERE CHARINDEX(column, 'abc') > 0
WHERE CAST(column AS DATETIME) <= '2010-01-01'
Anywhere you are manipulating table data--especially changing the data type--will render an index useless because an index is of the unaltered values and there's no way to relate to altered data.
For like looks to this article:
SQL Performance - Indexes and the LIKE clause
For NOT operator using indexes will depend on particular query.
From within Query Analyzer is an option called "Show Execution Plan" (located on the Query drop-down menu). If you turn this option on, then whenever you run a query in Query Analyzer, you will get a query execution plan. Use this to analyze the effectiveness of your query. Based on the results, you may need to use another query or add better indexes. E.g. table scan means that indexes are not used, bookmark lookups mean you should limit the rows or use a covering index, if there's a filter you might want to remove any function calls from the where clause, sort can be slow.
The search term you need to explore this issue in more depth is sargable.
here is one article to get you started:
http://www.sql-server-performance.com/tips/t_sql_where_p2.aspx
my query returns a column that can hold types of real estate. Values can be condo or duplex or house and so on. Instead of displaying condo, I just want a C in the column. My plan was to use a huge case/when structure to cover all the cases, is there an easier way? Just displaying the first letter in upper case wont work by the way, because sometimes that rule cant be applied to create the short code. Duplex for example is DE...
Thanks :-)
If you don't want to use a CASE statement how about creating a lookup table to map the column value to the lookup code you want. Join on to this in your query.
NB - Only worth considering if your query is running over a fairly small resultset or you'll hit performance issues. Indexing the column would help.
Some other options are depending on your DB server features:
Use a UDF to do the conversion.
Computed column on the source table.
Helper table with a column for each shorthand matching the long string?
The obvious thing to do would be to have another table which maps your value to a code, to which you can then join your results. But it smells a bit wrong. I'd want to join this other table to key values, not strings (which I assume aren't key values)
Why dont you use a decode function in sql
select decode(your_column_name,"condo","C",your_column_name) from table
I found one big issue.
I have added the Lower function to indexed column of one of the table to fetch the data.
The table contains more than 100K records.
While fetching the records, the cpu usage goes to 100%.
I could not understand, how this much drastic change can happen just because of Lower() function.
Please Help!
What you could do, if you really need this query a lot, is create a persisted computed column that uses the LOWER() function. Index that column and you should be fine again:
ALTER TABLE dbo.YourTableName
ADD LowerFieldName AS LOWER(YourFieldName) PERSISTED
CREATE NONCLUSTERED INDEX IX_YourTableName_LowerFieldName
ON dbo.YourTableName(YourFieldName)
That would keep a lower-case representation of your field in your table, it's always up to date, and since it's persisted, it's part of your table and doesn't incur the penalty of the LOWER() function. Put an index on it and your search should be as fast as it was before.
Links:
MSDN docs on SQL Server 2008 computed columns
Complex Computed Columns
Using Computed Columns in SQL Server with Persisted Values
Top 10 Hidden Gems in SQL Server 2005 - persisted computed columns are item #3
SQL Server Computed Columns
Working with computed columns
When you add LOWER() (or any function) around a column it is no longer possible to use an index (it is no longer SARG-able).
By default, SQL Server is not case sensitive, so you should be able to remove it.
I believe SQL Server is not case sensitive, so should remove it the Lower function, It should behave normally.
I have a column in a non-partitioned Oracle table defined as VARCHAR2(50); the column has a standard b-tree index. I was wondering if there is an optimal way to query this column to determine whether it contains a given value. Here is the current query:
SELECT * FROM my_table m WHERE m.my_column LIKE '%'||v_value||'%';
I looked at Oracle Text, but that seems like overkill for such a small column. However, there are millions of records in this table so looking for substring matches is taking more time than I'd like. Is there a better way?
No.
That query is a table scan. If v_value is an actual word, then you may very well want to look at Oracle Text or a simple inverted index scheme you roll your on your own. But as is, it's horrible.
Oracle Text covers a number of different approaches, not all of them heavyweight. As your column is quite small you could index it with a CTXCAT index.
SELECT * FROM my_table m
WHERE catsearch(m.my_column, v_value, null) > 0
/
Unlike the other type of Text index, CTXCAT indexes are transactional, so they do not require synchronisation. Such indexes consume a lot of space, but that you have to pay some price for improved performance.
Find out more.
You have three choices:
live with it;
use something like Oracle Text for full-text searching; or
redefine the problem so you can implement a faster solution.
The simplest way to redefine the problem is to say the column has to start with the search term (so lose the first %), which will then use the index.
An alternative way is to say that the search starts on word boundaries (so "est" will match "estimate" but not "test"). MySQL (MyISAM) and SQL Server have functions that will do matching like this. Not sure if Oracle does. If it doesn't you could create a lookup table of words to search instead of the column itself and you could populate that table on a trigger.
You could put a function-based index on the column, using the REGEXP_LIKE function. You might need to create the fbi with a case statement to return '1' with a match, as boolean returning functions dont seem to be valid in fbi.
Here is an example.
Create the index:
CREATE INDEX regexp_like_on_myCol ON my_table (
CASE WHEN REGEXP_LIKE(my_column, '[static exp]', 'i')
THEN 1
END);
And then to use it, instead of:
SELECT * FROM my_table m WHERE m.my_column LIKE '%'||v_value||'%';
you will need to perform a query like the following:
SELECT * FROM my_table m WHERE (
CASE WHEN REGEXP_LIKE(m.my_column, '[static exp]', 'i')
THEN 1
END) IS NOT NULL;
A significant shortcomming in this approach is that you will need to know your '[static exp]' at the time that you create your index. If you are looking for a performance increase while performing ad hoc queries, this might not be the solution for you.
A bonus though, as the function name indicates, is that you have the opportunity to create this index using regex, which could be a powerful tool in the end. The evaluation hit will be taken when items are added to the table, not during the search.
You could try INSTR:
...WHERE INSTR(m.my_column, v_value) > 0
I don't have access to Oracle to test & find out if it is faster than LIKE with wildcarding.
For the most generic case where you do not know in advance the string you are searching for then the best access path you can hope for is a fast full index scan. You'd have to focus on keeping the index as small as possible, which might have it's own problems of course, and could look at a compressed index if the data is not very high cardinality.