Working on speeding up a query and I've noticed for some reason the more empty columns added to a query the slower it gets.
With only the Id column the query returns 100k records in approx. 1 second.
If I add about 20 empty columns it goes to 4 seconds.
Questions
- What is the default data type of the string in SQL?
- Any way to speed this up?
SELECT Id,
'' as col1,
'' as col2,
'' as col3
FROM myTable
It will depend on how many rows are in your myTable. For ex: If you have 905k rows on mytable, Basically SQL is creating 20 Columns with ' ' for 905k rows
I just tried it in my own table that has 805k rows. For every increment of Columns I add, SQL creates '' values for each row.
Hope this helps you understand it clearer.
The default data type seems to be a varchar(1) -- you can insert it into a temp table and check the temp table structure to confirm. One option you can try is declare a variable and use it rather than the empty spaces:
declare #space varchar(40) = ''
SELECT
id,
#space as col1,
#space as col2,
#space as col3
FROM dbo.[table]
Related
How to Select a Column's Value Dynamically from a Table, where columns are not fixed?
I have a table name EMP which contains a numbers of dynamic columns.
id Items Col1 Col2 Col3 Col4 ... .... ... Col20 -----> Columns
1 ABC 20 40 100 32 .. .. . 200 -----> Values
Now I have a Dynamic Formula like
Round(Col1+Col4+Col20,2)+100
For Solving this formula I have to fetch the particular columns's value, which are associated to that columns and finally execute the query according to the formula.
I can Apply 2 methods
First Method
select col1,col4,col20 from EMP
but it will not work because I do not know the columns's name, even I get extract the required columns form the formula, but after getting the columns name. how can i select them dynamically from the EMP table?
****Second Method****
declare #holder int,#holder2 int.....
SELECT #holder=col1,#holder=col2..... FROM emp
It will also not work, because how can I Declare the a numbers of Dynamic variable for every column?
How can I do it?
Please Help
This is a very dubious requirement. Tables shouldn't have dynamic columns.
You can do something horrible with XML though (DB Fiddle)
The below will dynamically find all columns in the table whose name starts with the prefix "Col" and then cast the contents to decimal and sum them.
SELECT t.*, (SELECT t.* FOR XML PATH('x'), TYPE).value('
let $sum := sum(
for $col in x/*
where substring(local-name($col), 1, 3) eq "Col"
return ($col cast as xs:decimal ?)
)
return $sum',
'decimal(20,10)') AS Sum
FROM YourTable t
Declare
#sql1 as nvarchar(MAX),
#para as nvarchar(Max),-----'Holds all the columns name, that will extract from formula like col1,col2'
#formula nvarchar(MAX),--- Holds the formula like Round(Col1+Col4+Col20,2)+100
#ABC float,
set #sql1=N'
select * into datafetch from (
SELECT '+ #para+', '+#formula+' as newcol'+' FROM Emp)abc'
exec(#sql1)
SELECT #ABC=newcol FROM datafetch
---#ABC variable holds the final calculated value from the formula
You should only use Dynamic SQL for this purpose.
like this :
declare #columns varchar(max) = 'Col1 , Col2 , Col3' ;
declare #sql varchar(mx) = 'SELECT '+ #columns +'FROM emp' ;
I am working on a sql database which will provide with data some grid. The grid will enable filtering, sorting and paging but also there is a strict requirement that users can enter free text to a text input above the grid for example
'Engine 1001 Requi' and that the result will contain only rows which in some columns contain all the pieces of the text. So one column may contain Engine, other column may contain 1001 and some other will contain Requi.
I created a technical column (let's call it myTechnicalColumn) in the table (let's call it myTable) which will be updated each time someone inserts or updates a row and it will contain all the values of all the columns combined and separated with space.
Now to use it with entity framework I decided to use a table valued function which accepts one parameter #searchQuery and it will handle it like this:
CREATE FUNCTION myFunctionName(#searchText NVARCHAR(MAX))
RETURNS #Result TABLE
( ... here come columns )
AS
BEGIN
DECLARE #searchToken TokenType
INSERT INTO #searchToken(token) SELECT value FROM STRING_SPLIT(#searchText,' ')
DECLARE #searchTextLength INT
SET #searchTextLength = (SELECT COUNT(*) FROM #searchToken)
INSERT INTO #Result
SELECT
... here come columns
FROM myTable
WHERE (SELECT COUNT(*) FROM #searchToken WHERE CHARINDEX(token, myTechnicalColumn) > 0) = #searchTextLength
RETURN;
END
Of course the solution works fine but it's kinda slow. Any hints how to improve its efficiency?
You can use an inline Table Valued Function, which should be quite a lot faster.
This would be a direct translation of your current code
CREATE FUNCTION myFunctionName(#searchText NVARCHAR(MAX))
RETURNS TABLE
AS RETURN
(
WITH searchText AS (
SELECT value token
FROM STRING_SPLIT(#searchText,' ') s(token)
)
SELECT
... here come columns
FROM myTable t
WHERE (
SELECT COUNT(*)
FROM searchText
WHERE CHARINDEX(s.token, t.myTechnicalColumn) > 0
) = (SELECT COUNT(*) FROM searchText)
);
GO
You are using a form of query called Relational Division Without Remainder and there are other ways to cut this cake:
CREATE FUNCTION myFunctionName(#searchText NVARCHAR(MAX))
RETURNS TABLE
AS RETURN
(
WITH searchText AS (
SELECT value token
FROM STRING_SPLIT(#searchText,' ') s(token)
)
SELECT
... here come columns
FROM myTable t
WHERE NOT EXISTS (
SELECT 1
FROM searchText
WHERE CHARINDEX(s.token, t.myTechnicalColumn) = 0
)
);
GO
This may be faster or slower depending on a number of factors, you need to test.
Since there is no data to test, i am not sure if the following will solve your issue:
-- Replace the last INSERT portion
INSERT INTO #Result
SELECT
... here come columns
FROM myTable T
JOIN #searchToken S ON CHARINDEX(S.token, T.myTechnicalColumn) > 0
I have a table with two columns, one is of type Varchar and the other in NVarchar.
I want to update all the rows so VarcharField = NVarcharField.
It won't let me because some of the rows contain chars that are not allowed in varchar column with the current code page.
How can I find these rows?
Is it possible to remove any char that doesn't fit the specific code page I'm using?
SQL Server 2012.
You can find the rows by attempting to convert the nvarchar() col to varchar():
select nvarcharcol
from t
where try_convert(varchar(max), nvarcharcol) is null;
Try this..
to find the rows with values that are not supported by varchar
declare #strText nvarchar(max)
set #strText = 'Keep calm and say தமிழன்டா'
select cast(#strText as varchar(max)) col1 , N'Keep calm and say தமிழன்டா' col2
Here #strText has non-english chars, When you try to cast that into varchar the non-english chars turns into ????. So the col1 and col2 are not equal.
select nvar_col
from tabl_name
where nvar_col != cast(nvar_col as varchar(max))
Is it possible to remove any char that doesn't fit the specific code page I'm using?
update tabl_name
set nvar_col = replace(cast(nvar_col as varchar(max)),'?','')
where nvar_col != cast(nvar_col as varchar(max))
Replace ? with empty string and update them.
If Gordon's approach doesn't work because you get question marks from TRY_CONVERT instead of the expected NULL, try this approach:
SELECT IsConvertible = CASE WHEN NULLIF(REPLACE(TRY_CONVERT(varchar(max), N'人物'), '?',''), '') IS NULL
THEN 'No' ELSE 'Yes' END
If you need it as filter for the rows that can't be converted:
SELECT t.*
FROM dbo.TableName t
WHERE NULLIF(REPLACE(TRY_CONVERT(varchar(max), t.NVarcharField), '?',''), '') IS NULL
I have this type of data in a column in my table,,,
QCIM1J77477, 4650125572, QCCR1J77891, 4650312729, QCCR1J74974 --- remove without comma
or
QCIM1E107498,QCIM1E109835,4650092399/ QCCR1E91190, -- remove 4650092399
I want only that string which starts from QC,remove apart from QC,
so please tell me how can I achive it?
Beneath a piece of t-sql script that creates a temporary table #t with temporary variables. Here the temporary table exists untill you break your session, temporary variables exist during the execution of the script. I have a drop table statement at the bottom. Figure out yourself what you want with the table data and whether you want the data put in somewhere else, for example in a not-temporary table :).
I assume you want all the pieces of the string that contain 'QC' as seperate values. If you want your data back as it was originally, that is multiple strings per one column, then you could also do a group by trick. Then you do need a unique identifier of some sort, like name, id, guid of each row or identity.
create table #t ([QCs] nvarchar(100))
declare #str nvarchar(500)
set #str = 'QCIM1E107498,QCIM1E109835,4650092399/ QCCR1E91190'
--replace the above temporary variable with the column you are selecting
declare #sql nvarchar(4000)
select #sql = 'insert into #t select '''+ replace(#str,',',''' union all select ''') + ''''
print #sql
exec ( #sql )
select
QCs
,PATINDEX('%QC%',QCs) as StartPosition
,SUBSTRING(QCs,PATINDEX('%QC%',QCs),12) as QCsNew
from #t where QCs like '%QC%'
drop table #t
With PATINDEX you find the position where in the string 'QC' starts, and with SUBSTRING you tell the dbms to give back (here) 12 characters starting from the found StartPosition.
Beneath what the result looks like. QCsNew is your desired result.
QCs StartPosition QCsNew
QCIM1E107498 1 QCIM1E107498
QCIM1E109835 1 QCIM1E109835
4650092399/ QCCR1E91190 13 QCCR1E91190
Scenario:
I have a table1(col1 char(5)); A value in table1 may '001' or '01' or '1'.
Requirement:
Whatever value in col1, I need to retrive it in 5 char length concatenate with leading '0' to make it 5 char long.
Technique I applied:
select right(('00000' + col1),5) from table1;
I didn't see any reason, why it doesn't work? but it didn't.
Can anyone help me, how I can achieve the desired result?
Since you're using a fixed width column, it's already of size 5 (with whitespace). You need to trim it:
DECLARE #table1 TABLE (col1 char(5))
INSERT INTO #table1 (col1) VALUES ('12345')
INSERT INTO #table1 (col1) VALUES ('1')
SELECT RIGHT('00000'+RTRIM(col1),5) FROM #table1
-- Output:
-- 12345
-- 00001
Or use varchar instead:
DECLARE #table2 TABLE (col1 varchar(5))
INSERT INTO #table2 (col1) VALUES ('12345')
INSERT INTO #table2 (col1) VALUES ('1')
SELECT RIGHT('00000'+col1,5) FROM #table2
-- Output:
-- 12345
-- 00001
If you are storing the data in a CHAR field you are probably getting right spaces buffered with blanks. e.g. 01 = "01 ". If your do a RIGHT("00000" + value, 5) it'll still be the original value. You need to do a RTRIM() on the value or store the data in a VARCHAR field.
The problem is that the char(5) field is always 5 characters long, not matter what you put into it. If you insert '01' into the field, the value stored is actually '01 ' (note the trailing spaces).
Try this:
select right(('00000' + replace(col1, ' ', '')), 5)
Edit: I will leave my answer here as an example, but Michael's answer using rtrim is better.
you need to store your data in a consistent manner, so you don't need to write queries to format the data each time. this will fix your existing data:
UPDATE table1
SET col1= RIGHT('00000'+ISNULL(RTRIM(col1),''),5)
now every time you select you only have to do this:
SELECT col1 FROM table1
however, you must make sure that the data is formatted properly (leading zeros) every time it is inserted. I'd add a check constraint just to make sure:
ALTER TABLE table1 ADD CONSTRAINT
CK_table1_col1 CHECK (LEN(col1)=5)
and when you insert do this:
INSERT INTO table1
(col1, ...
VALUES
(RIGHT('00000'+ISNULL(RTRIM(#col1),''),5)