Get fill rates in another table - SQL Server - sql

Question

I am trying to create a script to calculate the fill rates for each column in a table Data_table and insert it into a second table Metadata_table.
The Data_table has 30 columns in it, and some columns have 100% data in them, and some have less than 100% (due to nulls).
My code to calculate the fill rate looks like this,
select
cast(sum(case
when employee_id is null
then 0
else 1
end) / cast(count(1) as float ) * 100 as decimal(8,3)) as employee_id_fill,
.....--/*so on for 30 columns..*/
from
[Data_table]
The Metadata_table should look like this:
Table_name | Colmn_name | Fill_rate
[Data_table]| Colomn_a | 100%
[Data_table]| Colomn_b | 89%
[Data_table]| Colomn_c | 100%
and so on...
I think
unpivot
can work here, but i am unable to get the column names into the [Metadata_table] automatically.
I tried using this for automating the column names-
COL_NAME(OBJECT_ID('DBO.[DATA_TABLE]'),'COLOMN_A')
but this has not worked so far.
Any help is appreciated

Answer 1

You can use sys.columns for grabbing the column names. You can join it to sys.tables by the object_id if you ever need to associate the two.
For example:
SELECT c.NAME
FROM SYS.TABLES t
INNER JOIN SYS.COLUMNS c ON t.OBJECT_ID = c.OBJECT_ID
WHERE t.OBJECT_ID = OBJECT_ID('DBO.[Data_Table]');
You can generate SQL from here in the format you wanted by creating an expression to query your table and then unpivot it.
Another approach could be a while loop to do inserts into your metadata table. If you're working with a very large table this option will be more expensive so keep it in mind. I used an example table dbo.Attendance_Records and this script will print out the example SQL, not execute it. You would want to change it to call sp_executesql on that text.
DECLARE #Table NVARCHAR(128) = 'DBO.[Attendance_Records]'
,#MetaTable NVARCHAR(128) = 'DBO.[Metadata_Table]'
,#ColumnName NVARCHAR(128)
,#Iterator INT = 1
,#SQL NVARCHAR(MAX)
SELECT c.NAME
,c.COLUMN_ID
,ROW_NUMBER() OVER (ORDER BY COLUMN_ID) AS RN
INTO #Cols
FROM SYS.COLUMNS c
WHERE c.OBJECT_ID = OBJECT_ID(#Table);
WHILE #Iterator <= (SELECT ISNULL(MAX(RN),0) FROM #Cols)
BEGIN
SET #ColumnName = (SELECT NAME FROM #Cols WHERE RN = #Iterator)
SET #SQL = 'INSERT INTO ' + #MetaTable + ' (Table_Name, Column_Name, Fill_Rate) '
+ 'SELECT ''' + REPLACE(#Table,'DBO.','') + ''', ''' + #ColumnName + ''', 100 * CONVERT(DECIMAL(8,3), SUM(CASE WHEN [' + #ColumnName + '] IS NULL THEN 0 ELSE 1 END)) / COUNT(1) AS [' + #ColumnName + '_fill]' + ' FROM ' + #Table
PRINT #SQL
SET #Iterator += 1
END

Answer 2

Since you need to have the column names you would need to so something along these lines.
select ColumnName = 'Colomn_a'
, FillRate = count(distinct Colomn_a) / count(*) * 1.0 --must multiply by 1.0 to avoid integer math
from YourTable
UNION ALL
select 'Colomn_b'
, count(distinct Colomn_b) / count(*) * 1.0
from YourTable

Answer 3

Just an alternate method of Mike R's
CREATE OR ALTER PROCEDURE [dbo].[GetFillRate_new] -- EXEC [GetFillRate] 'TestEmp'
(
#TableName NVARCHAR(128),
#Include_BlankAsNotFilled BIT = 1 -- 0-OFF(Default); 1-ON(Blank As Not Filled Data)
)
AS
BEGIN
SET NOCOUNT ON;
SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;
IF NOT EXISTS(SELECT 1 FROM SYS.OBJECTS WHERE [TYPE]='U' AND [NAME]=#TableName )
BEGIN
SELECT Result = 1 , Reason ='Table not exists in this Database' ;
RETURN 1;
END;
declare #sql varchar(max)
set #sql=''
select
#sql=#sql+'select
'''+c.column_name+''' as [Column Name],
cast((100*(sum(
case when ' +
case
when #include_blankasnotfilled = 0
then '[' + c.column_name + '] is not null'
when c.collation_name is null
then '[' + c.column_name + '] is not null'
else 'isnull([' + c.column_name + '],'''')<>'''' ' end +
' then 1 else 0 end)*1.0 / count(*)))
as decimal(5,2)) as [Fill Rate (%)]
from '+c.table_name+'
union all '
from
information_schema.columns as c
inner join information_schema.tables as t
on c.table_name=t.table_name
where
t.table_type='base table' and
t.table_name =#tablename
set #sql=left(#sql,len(#sql)-10)
--print #sql
exec(#sql)
end
You can find more details into this blog post https://exploresql.com/2019/12/14/how-to-find-fill-rate-in-a-table/

Answer 4

Challenges:
The Schema changes like below things makes our Fill Rate approach little difficult than actual.
Table name changes
Column name changes
Data type changes
Removing Existing columns
Adding New Columns
Due to the above challenges, we cannot simply go for Static Solution to find Fill Rate of a table. Instead, we need something like Dynamic Approach to avoid our future re-works.
Prerequisite:
In the below sample, we are going to use one stored procedure named ‘Get_FillRate’ for demo. If any one have the same object name in database, please make sure to change the below stored procedure name.
Sample Table Creation with Data Loading Script
--dropping temp table if exists
IF OBJECT_ID('TempDb..#TestEmp') IS NOT NULL
DROP TABLE #TestEmp;
CREATE TABLE #TestEmp
(
[TestEmp_Key] INT IDENTITY(1,1) NOT NULL,
[EmpName] VARCHAR(100) NOT NULL,
[Age] INT NULL,
[Address] VARCHAR(100) NULL,
[PhoneNo] VARCHAR(11) NULL,
[Inserted_dte] DATETIME NOT NULL,
[Updated_dte] DATETIME NULL,
CONSTRAINT [PK_TestEmp] PRIMARY KEY CLUSTERED
(
TestEmp_Key ASC
)
);
GO
INSERT INTO #TestEmp
(EmpName,Age,[Address],PhoneNo,Inserted_dte)
VALUES
('Arul',24,'xxxyyy','1234567890',GETDATE()),
('Gokul',22,'zzzyyy',NULL,GETDATE()),
('Krishna',24,'aaa','',GETDATE()),
('Adarsh',25,'bbb','1234567890',GETDATE()),
('Mani',21,'',NULL,GETDATE()),
('Alveena',20,'ddd',NULL,GETDATE()),
('Janani',30,'eee','',GETDATE()),
('Vino',26,NULL,'1234567890',GETDATE()),
('Madhi',25,'ggg',NULL,GETDATE()),
('Ronen',25,'ooo',NULL,GETDATE()),
('Visakh',25,'www',NULL,GETDATE()),
('Jayendran',NULL,NULL,NULL,GETDATE());
GO
SELECT [TestEmp_Key],[EmpName],[Age],[Address],[PhoneNo],[Inserted_dte],[Updated_dte] FROM #TestEmp;
GO
Temp Table - #TestEmp
SQL Procedure For Finding Fill Rate in a Table - Dynamic Approach
Input Parameters
Both of the Input Parameters are mandatory.
#p_TableName - Data type used for this input Parameter is NVARCHAR(128) and Nullability is NOT NULL.
#p_Include_BlankAsNotFilled - Data type used for this input Parameter is BIT and Nullability is NOT NULL and either 0 or 1 needs to give. 0 is by Default and 0 means OFF. 1 is ON (when given as 1 - Blank entries will be considered As Not Filled Data).
Output Columns
There are Two output Columns. both of those are Non Nullable Output Columns.
[Column Name] - Data type used for this Output Column is sysname and Nullability is NOT NULL. All the Column Names for the user given Table Name would come as row values.
[Fill Rate (%)] - Data type used for this Output Column is DECIMAL(5,2) and Nullability is NOT NULL. Values from 0.00 to 100.00 would come in result with respective Column Names.
Info reg Stored Procedure
Created the store Procedure named - 'Get_FillRate'.
To avoid the number of rows returned, set NOCOUNT as ON.
Try, Catch Blocks are added for error handling's.
To read Uncommitted Modifications, set TRANSACTION ISOLATION LEVEL as READ UNCOMMITTED.
Parameter Sniffing Concept is also included.
Some handling's done on the Table Name input parameters to support user typing table name formats like '.table_name','..table_name','...table_name','table_name','[table_name]','dbo.table_name','dbo.[table_name]','[dbo].[table_name]' etc.,
Validation is included at the start, when user gives other than 'table name', stored procedure would throw 'Table not exists in this Database' as error message.
System table named SYS.OBJECTS and SYS.COLUMNS and System View named INFORMATION_SCHEMA.COLUMNS are used inside the stored procedure.
ORDINAL_POSITION from INFORMATION_SCHEMA.COLUMNS is used, to return the result set with the same column order that the table structure already has.
COLLATION_NAME from INFORMATION_SCHEMA.COLUMNS is used, to support conditions like blank is either need to consider or not, as not filled entries.
COLUMN_NAME from INFORMATION_SCHEMA.COLUMNS is used, to show the final result set with respective fill rates.
Dynamic Query is used, to support dynamic approach and this would avoid all the challenges that would come in static solutions like schema changes.
Both Method 1(Dynamic Query with WHILE LOOP) and Method 2(Dynamic Query with UNION ALL) produces same result sets and carries same functionality where some metrics like CPU time, Elapsed Time, Logical reads that are better in Method 2.
Method 1 - With the use of WHILE Loop
CREATE OR ALTER PROCEDURE [dbo].[Get_FillRate]
(
#p_TableName NVARCHAR(128),
#p_Include_BlankAsNotFilled BIT = 0 -- 0-OFF(Default); 1-ON(Blank As Not Filled Data)
)
AS
BEGIN
BEGIN TRY
SET NOCOUNT ON;
SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;
--Parameter Sniffing
DECLARE #TableName NVARCHAR(128),
#Include_BlankAsNotFilled BIT,
#ColumnName NVARCHAR(128),
#R_NO INT,
#DataType_Field BIT,
#i INT, --Iteration
#RESULT NVARCHAR(MAX);
SELECT #TableName = #p_TableName,
#Include_BlankAsNotFilled = #p_Include_BlankAsNotFilled,
#i = 1;
--To Support some of the table formats that user typing.
SELECT #TableName =REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(#TableName,'[',''),']',''),'dbo.',''),'...',''),'..',''),'.','');
--validation
IF NOT EXISTS(SELECT 1 FROM SYS.OBJECTS WHERE [TYPE]='U' AND [NAME]=#TableName )
BEGIN
SELECT Result = 1 , Reason ='Table not exists in this Database' ;
RETURN 1;
END;
--dropping temp table if exists - for debugging purpose
IF OBJECT_ID('TempDb..#Temp') IS NOT NULL
DROP TABLE #Temp;
IF OBJECT_ID('TempDb..#Columns') IS NOT NULL
DROP TABLE #Columns;
--temp table creations
CREATE TABLE #Temp
(
[R_NO] INT NOT NULL,
[ColumnName] NVARCHAR(128) NOT NULL,
[FillRate] DECIMAL(5,2) NOT NULL
PRIMARY KEY CLUSTERED (ColumnName)
);
CREATE TABLE #Columns
(
[R_NO] INT NOT NULL,
[Name] [sysname] NOT NULL,
[DataType_Field] BIT NOT NULL
PRIMARY KEY CLUSTERED ([Name])
);
INSERT INTO #Columns ([R_NO],[Name],[DataType_Field])
SELECT
COLUMN_ID,
[Name],
IIF(collation_name IS NULL,0,1)
FROM SYS.COLUMNS WHERE OBJECT_ID = OBJECT_ID(#TableName);
WHILE #i <= ( SELECT MAX(R_NO) FROM #Columns) --Checking of Iteration till total number of columns
BEGIN
SELECT #DataType_Field=DataType_Field,#ColumnName=[Name],#R_NO=[R_NO] FROM #Columns WHERE R_NO = #i;
SET #RESULT =
'INSERT INTO #Temp ([R_NO],[ColumnName], [FillRate]) ' +
'SELECT ' + QUOTENAME(#R_NO,CHAR(39)) + ',
''' + #ColumnName + ''',
CAST((100*(SUM(
CASE WHEN ' +
CASE
WHEN #Include_BlankAsNotFilled = 0
THEN '[' + #ColumnName + '] IS NOT NULL'
WHEN #DataType_Field = 0
THEN '[' + #ColumnName + '] IS NOT NULL'
ELSE 'ISNULL([' + #ColumnName + '],'''')<>'''' ' END +
' THEN 1 ELSE 0 END)*1.0 / COUNT(*)))
AS DECIMAL(5,2))
FROM ' + #TableName;
--PRINT(#RESULT); --for debug purpose
EXEC(#RESULT);
SET #i += 1; -- Incrementing Iteration Count
END;
--Final Result Set
SELECT
ColumnName AS [Column Name],
FillRate AS [Fill Rate (%)]
FROM #TEMP
ORDER BY [R_NO];
RETURN 0;
END TRY
BEGIN CATCH --error handling even it is fetching stored procedure
SELECT
ERROR_NUMBER() AS ErrorNumber
,ERROR_SEVERITY() AS ErrorSeverity
,ERROR_STATE() AS ErrorState
,ERROR_PROCEDURE() AS ErrorProcedure
,ERROR_LINE() AS ErrorLine
,ERROR_MESSAGE() AS ErrorMessage;
RETURN 1;
END CATCH;
END;
Execute this stored procedure - Method 1 by passing the table name like below
Execute like below if we need to consider NULL values alone as not filled
EXEC [Get_FillRate] #p_TableName='#TestEmp',#p_Include_BlankAsNotFilled=0;
Execute like below if we need to consider both NULL values and empty/blank values as not filled
EXEC [Get_FillRate] #p_TableName='#TestEmp',#p_Include_BlankAsNotFilled=1;
Method 1 -Output
Method 2 - With the use of UNION ALL
CREATE OR ALTER PROCEDURE [dbo].[Get_FillRate]
(
#p_TableName NVARCHAR(128),
#p_Include_BlankAsNotFilled BIT = 0 -- 0-OFF(Default); 1-ON(Blank As Not Filled Data)
)
AS
BEGIN
BEGIN TRY
SET NOCOUNT ON;
SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;
--Parameter Sniffing
DECLARE #TableName NVARCHAR(128),
#Include_BlankAsNotFilled BIT,
#RESULT NVARCHAR(MAX);
SELECT #TableName = #p_TableName,
#Include_BlankAsNotFilled = #p_Include_BlankAsNotFilled,
#RESULT = '';
--To Support some of the table formats that user typing.
SELECT #TableName =REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(#TableName,'[',''),']',''),'dbo.',''),'...',''),'..',''),'.','');
--validation
IF NOT EXISTS(SELECT 1 FROM SYS.OBJECTS WHERE [TYPE]='U' AND [NAME]=#TableName )
BEGIN
SELECT Result = 1 , Reason ='Table not exists in this Database' ;
RETURN 1;
END;
--dropping temp table if exists - for debugging purpose
IF OBJECT_ID('TempDb..#Columns') IS NOT NULL
DROP TABLE #Columns;
--temp table creations
CREATE TABLE #Columns
(
[ORDINAL_POSITION] INT NOT NULL,
[COLUMN_NAME] [sysname] NOT NULL,
[DataType_Field] BIT NOT NULL,
[TABLE_NAME] [sysname] NOT NULL
PRIMARY KEY CLUSTERED ([ORDINAL_POSITION],[COLUMN_NAME])
);
INSERT INTO #Columns ([ORDINAL_POSITION],[COLUMN_NAME],[DataType_Field],[TABLE_NAME])
SELECT
[ORDINAL_POSITION],
[COLUMN_NAME],
CASE WHEN COLLATION_NAME IS NOT NULL THEN 1 ELSE 0 END,
[TABLE_NAME]
FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME =#tablename; --Using System_View
--Final Result Set
SELECT #RESULT = #RESULT+ N'SELECT '''+C.COLUMN_NAME+''' AS [Column Name],
CAST((100*(SUM(
CASE WHEN ' +
CASE
WHEN #include_blankasnotfilled = 0
THEN '[' + C.COLUMN_NAME + '] IS NOT NULL'
WHEN C.[DataType_Field]=0
THEN '[' + C.COLUMN_NAME + '] IS NOT NULL'
ELSE 'ISNULL([' + C.COLUMN_NAME + '],'''')<>'''' ' END +
' THEN 1 ELSE 0 END)*1.0 / COUNT(*)))
AS DECIMAL(5,2)) AS [Fill Rate (%)]
FROM '+C.TABLE_NAME+' UNION ALL '
FROM #Columns C;
SET #RESULT=LEFT(#RESULT,LEN(#RESULT)-10); --To Omit 'Last UNION ALL '.
--PRINT(#RESULT); --for debug purpose
EXEC(#RESULT);
RETURN 0;
END TRY
BEGIN CATCH --error handling even it is fetching stored procedure
SELECT
ERROR_NUMBER() AS ErrorNumber
,ERROR_SEVERITY() AS ErrorSeverity
,ERROR_STATE() AS ErrorState
,ERROR_PROCEDURE() AS ErrorProcedure
,ERROR_LINE() AS ErrorLine
,ERROR_MESSAGE() AS ErrorMessage;
RETURN 1;
END CATCH;
END;
Execute this stored procedure - Method 2 by passing the table name like below
Execute like below if we need to consider NULL values alone as not filled
EXEC [Get_FillRate] #p_TableName='#TestEmp',#p_Include_BlankAsNotFilled=0;
Execute like below if we need to consider both NULL values and empty/blank values as not filled
EXEC [Get_FillRate] #p_TableName='#TestEmp',#p_Include_BlankAsNotFilled=1;
Method 2 -Output
Metrics Difference between Method 1 Vs Method 2
The below four metrics taken for consideration for knowing the difference between Method 1 Vs Method 2
No. of Query Sets in Exec Query Plan
Total CPU Time (in ms)
Total Elapsed Time (in ms)
Total Logical Reads
In conclusion, we have seen how to find the Fill Rate of a Table Using T-SQL Queries that is applicable to run in both AZURE and On-Premises SQL Databases. Thus, it would helps us to take business decisions effectively as well as immediately.