What is the most efficient way to search a column and return all matching values while ignoring special characters?
For example if a table has a part_number column with the following values '10-01' '14-02-65' '345-23423' and the user searches for '10_01' and 140265 it should return '10-01' and '14-02-65'.
Processing the input to with a regex to remove those characters is possible, so the stored procedure could could be passed a parameter '1001 140265' then it could split that input to form a SQL statement like
SELECT *
FROM MyTable
WHERE part_number IN ('1001', '140265')
The problem here is that this will not match anything. In this case the following would work
SELECT *
FROM MyTable
WHERE REPLACE(part_number,'-','') IN ('1001', '140265')
But I need to remove all special characters. Or at the very least all of these characters ~!##$%^&*()_+?/\{}[]; with a replace for each of those characters the query takes several minutes when the number of parts in the IN clause is less than 200.
Performance is improved by creating a function that does the replaces, so the query takes less than a minute. But without removals the query takes around 1 second, is there any way to create some kind of functional index that will work on multiple SQL Server engines?
You could use a computed column and index it:
CREATE TABLE MyTable (
part_number VARCHAR(10) NOT NULL,
part_number_int AS CAST(replace(part_number, '-', '') AS int)
);
ALTER TABLE dbo.MyTable ADD PRIMARY KEY (part_number);
ALTER TABLE dbo.MyTable ADD UNIQUE (part_number_int);
INSERT INTO dbo.MyTable (part_number)
VALUES ('100-1'), ('140265');
SELECT *
FROM dbo.MyTable AS MT
WHERE MT.part_number_int IN ('1001', '140265');
Of course your replace statement will be more complex and you'll have to sanitize user input the same way you sanitize column values. But this is going to be the most efficient way to do it.
This query can now seek your column efficiently:
But to be honest, I'd just create a separate column to store cleansed values for querying purpose and keep the actual values for display. You'll have to take care of extra update/insert clauses, but that's a minimum damage.
Related
In my assignment with Oracle 11g, I am asked to make a table with column has this structure:
[NL|TE|][0-9]^10
Where NL or TE is inputed when INSERT row and [0-9]^10 is an auto-increment 10 digits number.
Example:
NL1234567890 or TE0253627576
When INSERT, the user should only write this:
INSERT INTO TableA VALUES ('NL');
And the DBMS take care of the rest. So how can I do so? Im still a newbie in this thing.
CREATE SEQUENCE your_seq;
/
CREATE OR REPLACE TRIGGER your_tablename_BI
BEFORE INSERT
ON your_tablename
REFERENCING NEW AS NEW OLD AS OLD
FOR EACH ROW
BEGIN
:NEW.your_col := :NEW.your_col || trim(to_char(your_seq.nextval, '0000000000'));
END your_tablename_BI;
/
Sample code?
'NL' || to_char(yoursequence.nextval)
I would keep them as separate columns. One is a VARCHAR2 that takes NL or whatever, the other is a NUMBER which is populated by the sequence.
You can then concatenate them at query time (put it in a view if you want) or use a virtual column.
Why? I can almost guarantee you that at some point you'll have a requirement to query the table on the character portion, or the numeric portion, or to sort on one or the other. Since you kept them separate, this is easy. If you had squashed them into a single column, you would have had to parse the values out at query time which leads to more complicated code than you need.
I want to write a query in T-SQL to perform a search on two concatenated columns. The two columns are fname and lname. Here is what I have so far:
SELECT
fname,
lname,
...
FROM
users
JOIN
othertable ON foo=bar
WHERE
fname+' '+lname LIKE '%query%'
SQL server doesn't like that syntax, though. How do I structure the query so that I can perform a WHERE LIKE operation that searches through two concatenated columns, allowing me to search the user's full name, rather than just first name and last name individually?
I can only suggest that one of fname or lname is NULL so the LIKE fails., (NULL concat anything is null)
Try
...
ISNULL(fname, '') + ' ' + ISNULL(lname, '') LIKE '%query%'
However, I would use a computed column and consider indexing it because this will run awfully.
My suggestion is to add a calculated column to your table for full_name
calculated column examples:
--drop table #test
create table #test (test varchar (10) , test2 varchar (5),[Calc] AS right(test, 3))
Insert #test
values('hello', 'Bye')
Insert #test
values('hello-bye', null)
Alter table #test
add [MyComputedColumn] AS substring(test,charindex('-',test), len(test)),
Concatenatedcolum as test+ ' ' +test2
select * from #test
As you can see you may have to play around a bit until you get the results you want. Do that in a temp table first to avoid having to restructure the database table multiple times. For names, especially if you are using middle name which is often blank, you may need to add some code to handle nulls. You may also need to have code sometimes to cast to the same datatype if one filed you are concatenating is an int for instance and the other a varchar.
I think one of the join conditions might be causing a problem. Try rewriting it, you may find the error goes away ;)
I have a transaction table of 1 million rows. The table has a field name "Code" to keep customer's ID. There are about 10,000 different customer code.
I have an GUI interface allow user to render a report from transaction table. User may select arbitrary number of customers for rendering.
I use IN operator first and it works for few customers:
SELECT * FROM TRANS_TABLE WHERE CODE IN ('...', '...', '...')
I quickly run into problem if I select few thousand customers. There is limitation using IN operator.
An alternate way is create a temporary table with only one field of CODE, and inject selected customer codes into the temporary table using INSERT statement. I may then using
SELECT A.* FROM TRANS_TABLE A INNER JOIN TEMP B ON (A.CODE=B.CODE)
This works nice for huge selection. However, there is performance overhead for temporary table creation, INSERT injection and dropping of temporary table.
Do you aware of better solution to handle this situation?
If you use SQL Server 2008, the fastest way to do this is usually with a Table-Valued Parameter (TVP):
CREATE TYPE CodeTable AS TABLE
(
Code int NOT NULL PRIMARY KEY
)
DECLARE #Codes AS CodeTable
INSERT #Codes (Code) VALUES (1)
INSERT #Codes (Code) VALUES (2)
INSERT #Codes (Code) VALUES (3)
-- Snip codes
SELECT t.*
FROM #Codes c
INNER JOIN Trans_Table t
ON t.Code = c.Code
Using ADO.NET, you can populate the TVP directly from your code, so you don't need to generate all those INSERT statements - just pass in a DataTable and ADO.NET will handle the rest. So you can write a Stored Procedure like this:
CREATE PROCEDURE GetTransactions
#Codes CodeTable READONLY
AS
SELECT t.*
FROM #Codes c
INNER JOIN Trans_Table t
ON t.Code = c.Code
... and just pass in the #Codes value as a parameter.
You can generate SQL such as
SELECT * FROM TRANS_TABLE WHERE CODE IN (?,?,?,?,?,?,?,?,?,?,?)
and re-use it in a loop until you've loaded all the IDs you need. The advantage is that if you only need a few IDs your DB doesn't need to parse all those in-clauses. If many IDs is a rare case then the performance hit may not matter. If you are not worried about the SQL parsing cache then you can limit the size of the in clause to the DB's actual limit, so that sometimes you don't need a loop and other times you do.
As you have to pass the IDs somehow, IN should be the fastest way.
MSDN mentions:
Including an extremely large number of values (many thousands) in an IN clause can consume resources and return errors 8623 or 8632. To work around this problem, store the items in the IN list in a table.
If you still can use IN and the query is to slow, you could try to adjust your indexes like using some covering index for your query. Looking up random values by the clustered index can be slow, because of the random disk I/O required. A covering index could reduce that problem.
If you really pass limit of IN and you create a temporary table, I don't expect the creation of the table to be a major problem, as long as you insert the values at once (not thousands of queries of course). Choose the method with the least overhead, like one of those mentioned here:
http://blog.sqlauthority.com/2008/07/02/sql-server-2008-insert-multiple-records-using-one-insert-statement-use-of-row-constructor/
Of course, if there is some static pattern in your IDs you could select by that (like in SPs or UDFs). If you get those thousands of IDs out of your database itself, instead of passing them back and forth, you could just store them or use a subquery...
Maybe you could pass the customer codes to a stored procedure comma separated and use the split sql function mentioned here: http://www.devx.com/tips/Tip/20009.
Then declare a scalar table where you insert the splitted values in and use an IN clause.
CREATE PROCEDURE prc_dosomething (
#CustomerCodes varchar(MAX)
)
AS
DECLARE #customercodetable table(code varchar(10)) -- or whatever length you require.
SET #customercodetable = UTILfn_Split(#CustomerCodes) -- see the article above for the split function.
-- do some magic stuff here :).
I have a query in MS Access which creates a table from two subqueries. For two of the columns being created, I'm dividing one column from the first subquery into a column from the second subquery.
The datatype of the first column is a double; the datatype of the second column is decimal, with scale of 2, but I want the second column to be a double as well.
Is there a way to force the datatype when creating a table through a standard make-table Access query?
One way to do it is to explicitly create the table before putting anything into it.
Your current statement is probably like this:
SELECT Persons.LastName,Orders.OrderNo
INTO Persons_Order_Backup
FROM Persons
INNER JOIN Orders
ON Persons.P_Id=Orders.P_Id
WHERE FirstName = 'Alistair'
But you can also do this:
----Create NewTable
CREATE TABLE NewTable(FirstName VARCHAR(100), LastName VARCHAR(100), Total DOUBLE)
----INSERT INTO NewTableusing SELECT
INSERT INTO NewTable(FirstName, LastName, Total)
SELECT FirstName, LastName,
FROM Person p
INNER JOIN Orders o
ON p.P_Id = o.P_Id
WHERE p.FirstName = 'Alistair'
This way you have total control over the column types. You can always drop the table later if you need to recreate it.
You can use the cast to FLOAT function CDBL() but, somewhat bizarrely, the Access Database Engine cannot handle the NULL value, so you must handle this yourself e.g.
SELECT first_column,
IIF(second_column IS NULL, NULL, CDBL(second_column))
AS second_column_as_float
INTO Table666
FROM MyTest;
...but you're going to need to ALTER TABLE to add your keys, constraints, etc. Better to simply CREATE TABLE first then use INSERT INTO..SELECT to populate it.
You can use CDbl around the columns.
An easy way to do this is to create an empty table with the correct field types and then to an Append-To query and Access will automatically convert the data to the destination field.
I had a similar situation, but I had a make-table query creating a field with NUMERIC datatype that I wanted to be short text.
What I did (and I got the idea from Stack) is to create the table with the field in question as Short Text, and at the same time build a delete query to scrub the records. I think it's funny that a DELETE query in access doesn't delete the table, just the records in it - I guess you have to use a DROP TABLE function for that, to purge a table...
Then, I converted my make-table query to an APPEND query, which I'd never done before... and I just added the running of the DELETE query to my process.
Thank you, Stack Overflow !
Steve
I add a '& ""' to the field I want to make sure are stored as text, and a ' *1 ' (as in multiplying the amount by 1) to the fields I want to store as numeric.
Seems to do the trick.
To get an Access query to create a table with three numeric output fields from input numeric fields, (it kept wanting to make the output fields text fields), had to combine several of the above suggestions. Pre-establish an empty output table with pre-defined output fields as integer, double and double. In the append query itself, multiply the numeric fields by one. It worked. Finally.
I have a table that looks a bit like this actors(forename, surname, stage_name);
I want to update stage_name to have a default value of
forename." ".surname
So that
insert into actors(forename, surname) values ('Stack', 'Overflow');
would produce the record
'Stack' 'Overflow' 'Stack Overflow'
Is this possible?
Thanks :)
MySQL does not support computed columns or expressions in the DEFAULT option of a column definition.
You can do this in a trigger (MySQL 5.0 or greater required):
CREATE TRIGGER format_stage_name
BEFORE INSERT ON actors
FOR EACH ROW
BEGIN
SET NEW.stage_name = CONCAT(NEW.forename, ' ', NEW.surname);
END
You may also want to create a similar trigger BEFORE UPDATE.
Watch out for NULL in forename and surname, because concat of a NULL with any other string produces a NULL. Use COALESCE() on each column or on the concatenated string as appropriate.
edit: The following example sets stage_name only if it's NULL. Otherwise you can specify the stage_name in your INSERT statement, and it'll be preserved.
CREATE TRIGGER format_stage_name
BEFORE INSERT ON actors
FOR EACH ROW
BEGIN
IF (NEW.stage_name IS NULL) THEN
SET NEW.stage_name = CONCAT(NEW.forename, ' ', NEW.surname);
END IF;
END
According to 10.1.4. Data Type Default Values no, you can't do that. You can only use a constant or CURRENT_TIMESTAMP.
OTOH if you're pretty up-to-date, you could probably use a trigger to accomplish the same thing.
My first thought is if you have the two values in other fields what is the compelling need for redundantly storing them in a third field? It flies in the face of normalization and efficiency.
If you simply want to store the concatenated value then you can simply create a view (or IMSNHO even better a stored procedure) that concatenates the values into a pseudo actor field and perform your reads from the view/sproc instead of the table directly.
If you absolutely must store the concatenated value you could handle this in two ways:
1) Use a stored procedure to do your inserts instead of straight SQL. This way you can receive the values and construct a value for the field you wish to populate then build the insert statement including a concatenated value for the actors field.
2) So I don't draw too many flames, treat this suggestion with kid gloves. Use only as a last resort. You could hack this behavior by adding a trigger to build the value if it is left null. Generally, triggers are not good. They add unseen cost and interactions to fairly simple interactions. You can, though, use the CREATE TRIGGER to update the actors field after a record is inserted or updated. Here is the reference page.
As of MySQL 8.0.13, you can use DEFAULT clause for a column which can be a literal constant or an expression.
If you want to use an expression then, simply enclose the required expression within parentheses.
(concat(forename," ",surname))
There are two ways to accomplish what you are trying to do as per my knowledge:
(important: consider backing up your table first before running below queries)
1- Drop the column "stage_name" all together and create a new one with DEFAULT constraint.
ALTER TABLE actors ADD COLUMN stage_name VARCHAR(20) DEFAULT (concat(forename," ",surname))
2- This will update newer entries in the column "stage_name" but not the old ones.
ALTER TABLE actors alter stage_name set DEFAULT (concat(forename," ",surname));
After that, if you need to update the previous values in the column "stage_name" then simply run:
UPDATE actors SET stage_name=(concat(forename," ",surname));
I believe this should solve your problem.