I have a static SQL query which finds out blank counts and data types (whether numeric or alphanumeric etc). Below is the query:
SELECT
case when Pattern = '' then 'BLANK' else Pattern end AS Pattern,
LEN(case when Pattern = '' then 'BLANK' else Pattern end) Length,
COUNT(*) AS Count FROM
( SELECT REPLACE(REPLACE( REPLACE( REPLACE( REPLACE(REPLACE( REPLACE( REPLACE( REPLACE(
REPLACE( REPLACE( REPLACE( REPLACE(REPLACE( REPLACE( REPLACE( REPLACE(REPLACE( REPLACE(
REPLACE( REPLACE(REPLACE( REPLACE(
REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE
(REPLACE(REPLACE(REPLACE(REPLACE(UPPER(col_name),'A','A'),'B','A') ,'C','A') ,'D','A')
,'E','A'),'F','A') ,'G','A') ,'H','A') ,'I','A'),'J','A') ,'K','A') ,'L','A') ,'M','A')
,'N','A') ,'O','A') ,'P','A')
,'Q','A'),'R','A') ,'S','A') ,'T','A') ,'U','A') ,'V','A') ,'W','A') ,'X','A') ,'Y','A')
,'Z','A'),'8','N'),'7','N'),'6','N'),'5','N'),'4','N'),'3','N')
,'2','N'),'1','N'),'0','N'),'9','N')
AS Pattern FROM table_name) A GROUP BY Pattern;
How to convert this to a dynamic SQL so that I can iterate over all columns of a table?
Edit
Lets say I have a below table
CREATE TABLE Data (
Column1 varchar(50),
Column2 varchar(50),
Column3 varchar(50),
Column4 varchar(50),
)
INSERT INTO Data
(Column1, Column2, Column3, Column4)
VALUES
(NULL, NULL, 'ABC123', 'abc '),
('xyz', NULL, 'MNO300', 'XYZ123 ')
Now I want a output like below:
Count_Pat_1 --> Count is of only one data type i.e. either Alphabetic or
Numeric
Count_Part_2 --> Count is of mixed data type i.e. contains both alphabet and
number
Col_name Count_Pat_1 Count_Pat_2 Blank
Column1 1 0 1
Column2 0 0 2
Column3 0 2 0
Column4 1 1 0
How can I generate the above table? Clearly I need to resort to Dynamic SQL where the entire sql string needs to be set as variable.
DECLARE #sql nvarchar(max)
SET #sql = N''
SELECT #sql = #sql + CASE WHEN #stm = N'' THEN N'' ELSE N',' END +
'('+
'''' + c.name + ''', '+
'CASE '+
'WHEN ' + QUOTENAME(c.name) + ' IS NULL THEN 1 ' +
'ELSE 0 ' +
'END, ' +
'CASE '+
'WHEN ' + QUOTENAME(c.name) + ' IS NOT NULL THEN LEN(' +
QUOTENAME(c.name) + ') - ** the above SQL script ***
FROM sys.columns c
JOIN sys.tables t ON c.object_id = t.object_id
WHERE t.name = 'Data'
But inserting the above SQL script at the place as indicated by ** will not be giving the output properly.
Hence any hints would be appreciated.
SQL Server version: 2016
With dynamic SQL the trick is to get it working as static SQL first and then convert it to dynamic SQL. Some notes:
To get multiple rows, 1 for each column consider using union all.
To get a count of a case you'll need to use sum.
I've used some possible logic to get a count of one data type or mixed data type but you may find you need more precise logic.
DECLARE #Sql NVARCHAR(MAX) = '';
SELECT #Sql = #Sql
+ CASE WHEN #Sql = '' THEN 'SELECT ' ELSE ' UNION ALL SELECT ' END
+ '''' + c.[name] + ''' ColumnName'
+ ', SUM(CASE WHEN ' + QUOTENAME(c.[name]) + ' IS NOT NULL AND ' + QUOTENAME(c.[name]) + ' NOT LIKE ''%[^0-9.]%'' OR ' + QUOTENAME(c.[name]) + ' NOT LIKE ''%[0-9.]%'' THEN 1 ELSE 0 END) [Count_Pat_1]'
+ ', SUM(CASE WHEN ' + QUOTENAME(c.[name]) + ' IS NOT NULL AND ' + QUOTENAME(c.[name]) + ' LIKE ''%[^0-9.]%'' AND ' + QUOTENAME(c.[name]) + ' LIKE ''%[0-9.]%'' THEN 1 ELSE 0 END) [Count_Pat_2]'
+ ', SUM(CASE WHEN ' + QUOTENAME(c.[name]) + ' IS NULL THEN 1 ELSE 0 END) [Blank]'
+ ' FROM Data'
FROM sys.columns c
JOIN sys.tables t ON c.object_id = t.object_id
WHERE t.[name] = 'Data';
-- print #Sql; -- For debugging
exec sp_executesql #Sql;
Returns as requested:
ColumnName | Count_Pat_1 | Count_Pat_2 | Blank
---------------------------------------------------
Column1 | 1 | 0 | 1
Column2 | 0 | 0 | 2
Column3 | 0 | 2 | 0
Column4 | 1 | 1 | 0
Related
So the goal is to make a SQL query for table A but the where clause is stored in table B.
here is example
table B:
BatchNumber | DataItem | Operator | TValue
12345 | Branch | = | 2700
12345 | StockType| = |S
12345 | PRCode | <> |AD
from this table I need to make below where statement
select *
from testdata.TableA
where Branch = 2700 and StockType = 'S' and PRCode <> 'AD'
here are the different methods I tried using to form a sql
declare #whereClause varchar(400)
declare #schema varchar(10) = 'testdata'
declare #Batch varchar(15) = 12345
select #whereClause = N' COALESCE(#whereClause + ' ' ,'') + trim(DataItem) + ' ' + trim(Operator) + ' ' +
case
when trim(TValue) LIKE '[A-Za-z]%' then '''' + trim(TValue) + '''' + ' AND'
when trim(TValue) = ' ' then '''' + '''' + ' AND'
else trim(TValue) + ' AND'
end
from' + #schema + '.TableB where BatchNumber =' + #Batch
--remove AND from the end of statement
SET #whereClause = SUBSTRING(#whereClause,0, LEN(#whereClause) - 3)
PRINT #whereClause
However this is giving me syntax error with single codes in the case statement.
Msg 102, Level 15, State 1, Line 4
Incorrect syntax near ' ,') + trim(DataItem) + '.
I need case statement to add single quote around string operators.
I also tried different method with FOR XML PATH(''), but with this single quotes are eliminated and "<>" signs are changed to "gtlt"
really appreciate any help.
It seems like some of your single-quotations may have to be quoted or you could use variables. For example: #whereClause = N' COALESCE(#whereClause + ' ' ,'')... may require #whereClause = N' COALESCE(#whereClause + '' '' ,'''')... or something on those lines. You could store there result of COALESCE(#whereClause + ' ' ,'') in a variable and then concat it with other variables in a similar way.
Alternatively, you could write a query in SQL Server like this:
select 'select * '
union all select 'from testdata.TableA '
union all select 'where 1 = 1 '
union all
select concat(
' and ',
dataitem, ' ', operator, ' ',
case
when tvalue like '%[0-9]%' then tvalue
else concat('''', tvalue, '''')
end
)
from test /* or tableB */
where batchnumber = 12345; /*optional where clause*/
You'll get the output like this:
select *
from testdata.TableA
where 1 = 1
and Branch = 2700
and StockType = 'S'
and PRCode <> 'AD'
Example:
https://dbfiddle.uk/?rdbms=sqlserver_2019&fiddle=12c3110a0d82e754bfc16f686779cddd
I'm trying to find a way to grab the count of nulls and blanks of every column in a table and put the results into a table. I found this but don't know how to push it to a table? Would also like to add in a Count(distinct fieldname) as well. Any help would be appreciated!
DECLARE #t nvarchar(max)
SET #t = N'SELECT '
SELECT #t = #t + 'sum(case when ' + c.name + ' is null or ' + c.name + ' = '''' then 1 else 0 end) "' + c.name + '",
sum(case when ' + c.name + ' is null then 0 else 1 end) "Non-Null Values for ' + c.name + '",'
FROM sys.columns c
WHERE c.object_id = object_id('TableName');
SET #t = SUBSTRING(#t, 1, LEN(#t) - 1) + ' FROM TableName;'
EXEC sp_executesql #t
**Edit
This provides the results in one row. Is there a way to do it so each field gets it's own row in a table?
FieldName CountofNullsBlanks
FieldA 0
FieldB 100
Here:
DECLARE #t nvarchar(max)
SET #t = N'SELECT '+convert(nvarchar(max),count(*))+' as [Distinct fieldnames],' -- added the distinct
SELECT #t = #t + 'sum(case when ' + c.name + ' is null or ' + c.name + ' = '''' then 1 else 0 end) "' + c.name + '",
sum(case when ' + c.name + ' is null then 0 else 1 end) "Non-Null Values for ' + c.name + '",'
FROM sys.columns c
WHERE c.object_id = object_id('YOURTABLE');
SET #t = SUBSTRING(#t, 1, LEN(#t) - 1) + 'into ##a FROM YOURTABLE;' --added the into ##a
EXEC sp_executesql #t
Notice the "into ##a" towards the end; This saves the result "table" to a global temp, which will be available until all sessions that accessed it finish. Careful with the name or it might cause conflicts in the database.
I have a Dynamic query which fetch a row with 62 columns in sql
The query is like this
DECLARE #SQLQuery AS nvarchar(max)
DECLARE #columns nvarchar(max)
SELECT
#columns = ISNULL(#columns + ', ', '') + QUOTENAME(Column_name)
FROM (SELECT
Column_name
FROM
(SELECT
Column_name,
ROW_NUMBER() OVER (PARTITION BY Column_name ORDER BY Column_name) AS Rows
FROM information_Schema.Columns
WHERE Table_Name IN ('Parking', 'AIV_Parking')
AND column_name NOT IN ('IssueNo', 'Issuedate', 'Agency', 'OfficerName', 'OfficerId', 'Beat', 'UNITSERIAL', 'VEHLICNO', 'VEHLICSTATE', 'VEHLICEXPDATE', 'VEHLICTYPE', 'VEHMAKE',
'VEHMODEL', 'VEHBODYSTYLE', 'VEHVIN4', 'VEHVIN', 'ISSUENOCHKDGT', 'VEHCOLOR1', 'VEHCOLOR2', 'PERMITNO', 'VEHLABELNO', 'LOCBLOCK', 'LOCSTREET', 'LOCDESCRIPTOR', 'LOCTRAVELDIR', 'LOCSIDEOFSTREET',
'LOCSUBURB', 'LOCLOT', 'LOCCROSSSTREET1', 'LOCCROSSSTREET2', 'LOCSTATE', 'LOCPOSTALCODE', 'METERNO', 'METERBAYNO', 'REMARK1', 'REMARK2')) p
WHERE Rows = 2) RequiredColumns
EXEC ('SELECT ' + #columns + ' FROM AIV_Parking Where Issueno =''100000600''')
Which gives me below records
So result is Something like below where i get only 1 row at a time.
|ACTLINE1 | ACTLINE2 | AUTOPROC_UNIQUEKEY | COURTADDR| IssueNo|.......
|SUBIACO | LOCAL LAWS| NULL | NULL | 123 |
Now i want this to converted like:
|FieldName | FieldValue
|ACTLINE1 | SUBIACO
|ACTLINE2 | LOCAL LAWS
|ISSUENO | 123
While pivoting these column ,I want only those columns which has value i don't want those column whose value is null or ' '
You need to use UNION ALL and filter the result with a WHERE clause:
DECLARE #SQLQuery AS NVARCHAR(MAX) = ''
SELECT #SQLQuery = #SQLQuery +
'SELECT ''' + COLUMN_NAME + ''' AS FieldName, CAST(' + QUOTENAME(COLUMN_NAME) + ' AS NVARCHAR(MAX)) AS FieldValue
FROM AIV_Parking
WHERE Issueno =''100000600''
UNION ALL' + CHAR(10)
--QUERY HERE TO GET COLUMNS FROM INFORMATION_SCHEMA.COLUMNS
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME IN('Parking','AIV_Parking')
IF #SQLQuery <> '' BEGIN
-- Remove the last occurence of UNION ALL
SELECT #SQLQuery = LEFT(#SQLQuery, LEN(#SQLQuery) - 11)
SELECT #SQLQuery =
'SELECT * FROM (' + #SQLQuery + ') t WHERE FieldValue IS NOT NULL AND FieldValue <> '''''
EXEC(#SQLQuery)
END
Note that you need to CAST your FieldValue to a NVARCHAR(MAX) or whatever datatype that will not produce a conversion error. This is because when using UNION all columns must have the same datatype, if not, they will be converted to the one with the higher data type precedence.
I'm a little new at SQL so please bear with me. I am attempting to write some a query that will allow me to loop through an entire table and find the number of times null values appear in each column. This is easy to do the hard way by typing the following:
Select
SUM(CASE COL_1 WHEN IS NULL THEN 1 ELSE 0 END) AS COL_1_NULLS
,SUM(CASE COL_2 WHEN IS NULL THEN 1 ELSE 0 END) AS COL_2_NULLS
FROM TABLE1
This is easy but it can become arduous if you want to do this for multiple tables or if a single table has a lot of columns.
I'm looking for a way to write a query that passes a table name into it and then loops through each column in the defined table (possibly pulling the column name by ordinance via a join to a metadata view?) and then sums the number of nulls in the column. Before anyone jumps on the nitpick bandwagon please keep in mind that this basic idea could be used for more than just finding nulls. Any assistance with this issue is greatly appreciated.
You need to use dynamic sql:
declare #custom_sql varchar(max)
set #custom_sql = 'SELECT null as first_row'
select
#custom_sql = #custom_sql + ', ' + 'SUM(CASE WHEN ' + COLUMN_NAME + ' IS NULL THEN 1 ELSE 0 END) as ' + COLUMN_NAME + '_NULLS'
from
INFORMATION_SCHEMA.COLUMNS where table_name = 'MYTABLE'
set #custom_sql = #custom_sql + ' FROM MYTABLE'
exec(#custom_sql)
You can also use the COALESCE term (just for a slightly different approach):
declare #custom_sql varchar(max)
select
#custom_sql = COALESCE(#custom_sql + ', ', '') + 'SUM(CASE WHEN ' + COLUMN_NAME + ' IS NULL THEN 1 ELSE 0 END) as ' + COLUMN_NAME + '_NULLS'
from
INFORMATION_SCHEMA.COLUMNS where table_name = 'users'
set #custom_sql = 'SELECT ' + #custom_sql
set #custom_sql = #custom_sql + ' FROM Users'
print #custom_sql
exec(#custom_sql)
I don't know how to make a generic query, but you can always generate the script like this
declare #sql nvarchar(max) = 'select 1 as dummy'
select #sql = #sql + '
, sum(case when [' + c.name + '] is null then 1 else 0 end) as [' + c.name + '_NULLS]'
from sys.columns c
join sys.tables t on t.object_id = c.object_id
where t.name = 'TABLE1'
set #sql = #sql + ' from TABLE1'
select #sql
Then you can execute the result eg. with exec sp_executesql #sql
For a cooler approach, you can use ISNULL to skip the first comma.
declare #sql nvarchar(max)
declare #tablename nvarchar(255) = 'xxxx'
Select #sql = ISNULL(#SQL + ',','') + ' ' + COLUMN_NAME + '_count = Sum(case when ' + COLUMN_NAME + ' is null then 1 else 0 end)' + char(13)
From information_schema.columns
where table_name = #tablename
set #sql = 'Select' + #sql + ' From ' + #tablename
print #sql
exec sp_executesql #sql
The following query returns the values of the table for each field in terms of null percentage . What I want is to get the sum of those percentages for a specific ProductID. Also, I would like to get a percentage (in an extra column) of the fields do not have value i.e. ="". Any ideas?
use AdventureWorks
DECLARE #TotalCount decimal(10,2), #SQL NVARCHAR(MAX)
SELECT #TotalCount = COUNT(*) FROM [AdventureWorks].[Production].[Product]
SELECT #SQL =
COALESCE(#SQL + ', ','SELECT ') +
'cast(sum (case when ' + QUOTENAME(column_Name) +
' IS NULL then 1 else 0 end)/#TotalCount*100.00 as decimal(10,2)) as [' +
column_Name + ' NULL %]
'
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'Product' and TABLE_SCHEMA = 'Production'
SET #SQL = 'set #TotalCount = NULLIF(#TotalCount,0)
' + #SQL + '
FROM [AdventureWorks].Production.Product'
print #SQL
EXECUTE SP_EXECUTESQL #SQL, N'#TotalCount decimal(10,2)', #TotalCount
You can use the following:
use AdventureWorks
DECLARE #colCount int;
DECLARE #nullCheck nvarchar(max) = N'';
DECLARE #emptyCheck nvarchar(max) = N'';
DECLARE #SQL NVARCHAR(MAX);
DECLARE #KeyToCheck int = 123; -- adapt as necessary
SELECT
#nullCheck += '
+ ' + 'count(' + QUOTENAME(column_Name) + ')'
,#emptyCheck += '
+ ' +
CASE
WHEN DATA_TYPE IN('bigint', 'int', 'smallint', 'tinyint', 'bit', 'money', 'smallmoney', 'numeric', 'decimal', 'float', 'real') THEN
-- check numeric data for zero
'sum(case when coalesce(' + QUOTENAME(column_Name) + ', 0) = 0 then 1 else 0 end)'
WHEN DATA_TYPE like '%char' or DATA_TYPE like '%text' THEN
--check character data types for empty string
'sum(case when coalesce(' + QUOTENAME(column_Name) + ', '''') = '''' then 1 else 0 end)'
ELSE -- otherwise, only check for null
'sum(case when ' + QUOTENAME(column_Name) + ' IS NULL then 1 else 0 end)'
END
,#colCount =
count(*) over()
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'Product' and TABLE_SCHEMA = 'Production'
;
SET #SQL = 'SELECT case when count(*) > 0 then 100.00 - (' + #nullCheck + '
) * 100.00 / ' + cast(#colCount as nvarchar(max)) + '.00 / count(*) end as null_percent
, case when count(*) > 0 then (' + #emptyCheck + '
) * 100.00 / ' + cast(#colCount as nvarchar(max)) + '.00 / count(*) end as empty_percent
FROM Production.Product
WHERE ProductID = ' + cast(#KeyToCheck as nvarchar(max))
;
print #SQL;
EXECUTE (#SQL)
I simplified one of your expressions: Instead of sum (case when <column> IS NULL then 1 else 0 end), you can just use count(<column>). When using count with an expression instead of *, it counts the rows where this expression is non-null. As this is the opposite from what you need, I added the 100.00 - as the start of the SELECT.
For the "empty check", this would make the logic more complex to understand, hence I left the original logic there and extended it. There, I implemented an check for emptiness for numeric and character/text data types. You can easily extend that for date, binary data etc. with whichever logic you use to determine if a column is empty.
I also found it more simple to leave first + in the two variables #nullCheck and #emptyCheck, as it is valid SQL to start an expression wit this.
I also extended the statement so that if there would potentially be more than one record with ProductId = 123, it shows the average across all records, i. e. the total sum divided by the count of rows. And the outermost case expressions just avoid an division by zero error if count(*) would be zero, i. e. no record with ProductId = 123 found. In that case the return value is null.
You could use AVG function:
SELECT AVG(CASE WHEN value IS NULL THEN 100 ELSE 0 END) AS Percents
FROM Table
UPDATE:
Here is your script:
DECLARE #SQL NVARCHAR(MAX), #TABLE_NAME NVARCHAR(MAX), #TABLE_SCHEMA NVARCHAR(MAX), #PK NVARCHAR(MAX)
SET #TABLE_NAME = 'tblBigTable'
SET #TABLE_SCHEMA = 'dbo'
SET #PK = '8'
SELECT
#SQL = COALESCE(#SQL + ', ', 'SELECT ') +'AVG(CASE WHEN ' + COLUMN_NAME + ' IS NULL THEN 100 ELSE 0 END) AS [' + COLUMN_NAME +' NULL %]'
FROM
INFORMATION_SCHEMA.COLUMNS
WHERE
TABLE_SCHEMA = #TABLE_SCHEMA AND
TABLE_NAME = #TABLE_NAME
SET #SQL = #SQL + ' FROM ' + #TABLE_NAME + ' WHERE pkId = ''' + #PK + ''''
print #SQL
EXECUTE SP_EXECUTESQL #SQL