I have a table, let's call it "table_X" in that table I have multiple columns (46) and in the future there is a possibility that we will expand it to have more columns, since the source of a table is an old ERP system, we need to transform the dataset in some cases, one of the transformation is that when we replace the '' values with NULLs and here is where I have problem, I wrote a dynamic update, because the previously mentioned reason (in the future we will have more columns), but I got error message and right now I am stuck.
DECLARE #SQL_columnnull NVARCHAR(max)
DECLARE #db2 NVARCHAR(max)
DECLARE #table2 NVARCHAR(max)
SET #db2 = 'db'
SET #table2 = 'table_X'
SELECT #SQL_columnnull_part_1 = STRING_AGG(CAST( N' UPDATE '+#db2+'.[dbo].'+#table2+' WITH (TABLOCK) SET ' AS NVARCHAR(MAX))
+QUOTENAME(COLUMN_NAME,'['']') + N' = NULL WHERE '
+QUOTENAME(COLUMN_NAME,'['']') ,+ N' = '''';')
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = #table2
AND INFORMATION_SCHEMA.COLUMNS.ORDINAL_POSITION <= 3
for the two first column, the code is able to populate the command parts properly, but when it reaches the last column then the "='';" won't be populated
UPDATE db.[dbo].table_X SET [Column_1] = NULL WHERE [Column_1] = '';
UPDATE db.[dbo].table_X SET [Column_2] = NULL WHERE [Column_2] = '';
UPDATE db.[dbo].table_X SET [Column_3] = NULL WHERE [Column_3]
You are messing a bit about with your STRING_AGG
The syntax is
STRING_AGG ( expression, separator )
Your separator is
+ N' = '''';'
Since the separator is not applied after the last entry, you get the result you see!
I would also be wary of the cast, you are casting the start of the expression as nvarchar(max), however you are the concatenationg non varchar strings.
Finally - why are you doing separate updates for each column? this is very poor performance!
First, a fixed query that does what you want would be:
SELECT
#SQL_columnnull =
STRING_AGG(
CAST(
' UPDATE ' + #db2 + '.[dbo].' + #table2 + ' WITH (TABLOCK) SET '
+ QUOTENAME(COLUMN_NAME, '['']') + N' = NULL WHERE '
+ QUOTENAME(COLUMN_NAME, '['']') +N' = '''''
AS NVARCHAR(MAX))
,';')
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = #table2
AND INFORMATION_SCHEMA.COLUMNS.ORDINAL_POSITION <= 3
I have included the full string within the cast, and the separator is now simply ";"
For performance I would however do this in stead:
SELECT
#SQL_columnnull =
N' UPDATE ' + #db2 + '.[dbo].' + #table2 + ' WITH (TABLOCK) SET ' +
STRING_AGG(
CAST(
QUOTENAME(COLUMN_NAME, '['']') + N'='+N'IIF('+ QUOTENAME(COLUMN_NAME, '['']') + N'= '''',NULL,'+ QUOTENAME(COLUMN_NAME, '['']')+') '
AS NVARCHAR(MAX))
,',
')
+'
WHERE '+
STRING_AGG(
CAST(
QUOTENAME(COLUMN_NAME, '['']') + N'= '''' '
AS NVARCHAR(MAX))
,' OR ')
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = #table2
AND INFORMATION_SCHEMA.COLUMNS.ORDINAL_POSITION <= 3
This does just one pass over your table, and updates the columns that have the wrong data.
Finally I would check that all the columns are of a varchar or nvarchar data type, to exclude columns of othe datattypes which might give errors
This gives
UPDATE db.[dbo].table_X
SET [Column_1] = IIF([Column_1] = '', NULL, [Column_1])
,[Column_2] = IIF([Column_2] = '', NULL, [Column_2])
,[Column_3] = IIF([Column_3] = '', NULL, [Column_3])
WHERE [Column_1] = ''
OR [Column_2] = ''
OR [Column_3] = ''
Related
When I attempt to execute the following DROP FUNCTION on Azure Synapse Serverless Pool I get the following error
Operation DROP FUNCTION is not allowed for a replicated database
The code that generates the error is:
DECLARE #FunctionScript nvarchar(max) = 'IF object_id(N''dbo.GetOptionSetLabel'') is not null DROP FUNCTION [dbo].[GetOptionSetLabel]'
Any thoughts on how to overcome the error?
I think I got marked down on this question because I didn't include the full code, so here it is...
USE [LakeDatabaseEnriched] --Specify the name of the database in which GetOptionSetLabel function will be created
DECLARE
#OptionSetLabelFunctionDatabase sysname, --It will be automatically be set with the name of the database that has GetOptionSetLabel function
#OptionSetLabelFunctionDatabaseSchema sysname, --Specify the name of the database in which GetOptionSetLabel function will be created
#OptionSetLabelFunctionDatabaseCollation varchar(256), --It will be automatically be set with the collation of the database that has GetOptionSetLabel function
#SynapseLinkDatabase sysname, --Specify the name of the database corresponding to your Synapse Link for Dataverse
#CurrentDatabaseCollation varchar(256)
SET #OptionSetLabelFunctionDatabase = QUOTENAME(DB_NAME())
SET #OptionSetLabelFunctionDatabaseCollation = CONVERT(varchar(256), DATABASEPROPERTYEX(DB_NAME(), 'collation'));
SET #OptionSetLabelFunctionDatabaseSchema = 'dbo'
SET #SynapseLinkDatabase = 'dataverse_xxxx_org5a2bcccf'
DECLARE #FunctionScript nvarchar(max) = 'IF object_id(N''dbo.GetOptionSetLabel'') is not null DROP FUNCTION [dbo].[GetOptionSetLabel]'
EXEC sp_executesql #FunctionScript
SET #FunctionScript = '
SET QUOTED_IDENTIFIER ON
SET ANSI_NULLS ON
'
EXEC sp_executesql #FunctionScript
SET #FunctionScript = '
CREATE FUNCTION [dbo].[GetOptionSetLabel]
(
#EntityName nvarchar(max), #ColumnName nvarchar(max), #Value nvarchar(max), #LanguageCode int
)
RETURNS
nvarchar(max)
AS
BEGIN
declare #Values table (
value nvarchar(max)
);
insert into #Values
SELECT Value
FROM STRING_SPLIT(#Value, '';'')
declare #Labels table(
AttributeValue int,
LabelText nvarchar(max)
);
insert into #Labels
select distinct *
from (
select gosm.[Option], gosm.LocalizedLabel Value
from '+#SynapseLinkDatabase+'.'+#OptionSetLabelFunctionDatabaseSchema+'.GlobalOptionsetMetadata gosm
inner join #Values v on cast(v.[Value] as int) = gosm.[Option]
where gosm.OptionSetName = #ColumnName
and gosm.LocalizedLabelLanguageCode = #LanguageCode
and TRY_CAST(v.[Value] as int) is not null
union
select osm.[Option], osm.LocalizedLabel Value
from '+#SynapseLinkDatabase+'.'+#OptionSetLabelFunctionDatabaseSchema+'.OptionsetMetadata osm
inner join #Values v on cast(v.[Value] as int) = osm.[Option]
where osm.OptionSetName = #ColumnName
and osm.EntityName = #EntityName
and osm.LocalizedLabelLanguageCode = #LanguageCode
and TRY_CAST(v.[Value] as int) is not null
) t
order by [Option] asc
declare #optionsetLabel nvarchar(max)
set #optionsetLabel = isnull(
(
select string_agg(LabelText, '', '')
from #Labels
),
#Value)
RETURN #optionsetLabel
END'
PRINT 'Beginning function creation'
--PRINT #FunctionScript
EXEC sp_executesql #FunctionScript
PRINT 'Completed function creation'
USE [dataverse_xxxx_org5a2bcccf] --Specify the name of the database corresponding to your Synapse Link for Dataverse to be used from here onward
--=================================================================================================
--PROVIDE INPUT PARAMETERS:
--=================================================================================================
DECLARE
#EnrichedViewDatabase sysname, --Specify the name of the database in which views with enriched entities will be created
#EnrichedViewSchema sysname, --Specify the name of the database schema in which views with enriched entities will be created
#EnrichedColumnSuffix varchar(50), --Specify the suffix for columns enriched with human-readable descriptions. For example, the suffix of "label" will change a statecode column in the base table to a statelabel column in the enriched view.
#LanguageCode varchar(10), --Specify the language code for localized labels. For example, English - United States is 1033 (https://learn.microsoft.com/en-us/openspecs/office_standards/ms-oe376/6c085406-a698-4e12-9d4d-c3b0ee3dbc4a)
#BaseTableSuffix varchar(50), --If applicable, specify the suffix in the names of the base tables or views (e.g., '_partitiond'). The default is an empty string.
#PreviewOnly bit --Indicate whether to preview the SQL Script (without creating the views) = 1 ; Create views = 0;
SET #EnrichedViewDatabase = 'LakeDatabaseEnriched'
SET #EnrichedViewSchema = 'dbo'
SET #EnrichedColumnSuffix = 'label'
SET #LanguageCode = 1033
SET #BaseTableSuffix = ''
SET #PreviewOnly = 0
SET #CurrentDatabaseCollation = CONVERT(varchar(256), DATABASEPROPERTYEX(DB_NAME(), 'collation'));
-- 'If still collation error occurs in script, search for Latin1_General_100_CI_AS_SC_UTF8 and replace it with getting collation from (SELECT #CurrentDatabaseCollation) query
--=================================================================================================
-- Do not edit the script below this point
--=================================================================================================
--Get column metadata from the Lake Database managed by Synapse Link for Dataverse
--The column metadata will be stored as a JSON document in a scalar variable
--This is needed as a workaround for the limitation of not allowing system objects to be used in distributed queries
DECLARE #ColumnMetadata nvarchar(MAX), #ColumnMetadataSQL nvarchar(MAX)
--Define the SQL statement to retrieve column metadata from the Lake Database managed by Synapse Link for Dataverse
--Results will be stored as a JSON document in a variable
SET #ColumnMetadataSQL = 'SET #ColumnMetadataOUT = (
SELECT TABLE_SCHEMA,
TABLE_NAME,
COLUMN_NAME,
ORDINAL_POSITION,
DATA_TYPE
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = ''dbo''
AND TABLE_NAME NOT IN (''OptionsetMetadata'', ''GlobalOptionsetMetadata'',''StateMetadata'',''StatusMetadata'', ''TargetMetadata'')
AND TABLE_NAME LIKE ''%' + (#BaseTableSuffix COLLATE Latin1_General_100_CI_AS_SC_UTF8) + '''
FOR JSON AUTO)'
DECLARE #ParmDefinition NVARCHAR(MAX);
SET #ParmDefinition = N'#ColumnMetadataOUT NVARCHAR(MAX) OUTPUT';
EXECUTE sp_executesql #ColumnMetadataSQL, #ParmDefinition, #ColumnMetadataOUT=#ColumnMetadata OUTPUT;
--Declare a variable to store a SQL statement for creating enriched views
DECLARE #SQL nvarchar(MAX) = ''
; WITH CM AS (
--Parse column metadata variable and construct a table based on its content
SELECT JSON_VALUE(CM.value, '$.TABLE_SCHEMA') AS TableSchema,
JSON_VALUE(CM.value, '$.TABLE_NAME') AS TableName,
LEFT(JSON_VALUE(CM.value, '$.TABLE_NAME'), LEN(JSON_VALUE(CM.value, '$.TABLE_NAME'))-LEN(#BaseTableSuffix)) AS EntityName,
JSON_VALUE(CM.value, '$.COLUMN_NAME') AS ColumnName,
CAST(JSON_VALUE(CM.value, '$.ORDINAL_POSITION') AS INT) AS OrdinalPosition,
JSON_VALUE(CM.value, '$.DATA_TYPE') AS DataType
FROM OPENJSON (#ColumnMetadata) AS CM
)
, OSM AS (
--Get Option Set Metadata
SELECT DISTINCT
EntityName,
OptionSetName,
QUOTENAME(EntityName + '_' + OptionSetName) AS Alias
FROM dbo.[OptionsetMetadata]
WHERE LocalizedLabelLanguageCode = #LanguageCode
)
, GOSM AS (
--Get Global Option Set Metadata
SELECT DISTINCT
OptionSetName,
QUOTENAME('Global_' + OptionSetName) AS Alias
FROM dbo.[GlobalOptionsetMetadata]
WHERE LocalizedLabelLanguageCode = #LanguageCode
)
, GOSMM AS (
--Get Global Option Set Metadata
SELECT DISTINCT
OptionSetName,
QUOTENAME('Global_Multiselect_' + OptionSetName) AS Alias
FROM dbo.[GlobalOptionsetMetadata]
WHERE LocalizedLabelLanguageCode = #LanguageCode
)
, StateM AS (
--Get State Metadata
SELECT DISTINCT
EntityName,
QUOTENAME(EntityName + '_State') AS Alias
FROM dbo.[StateMetadata]
WHERE LocalizedLabelLanguageCode = #LanguageCode
)
, StatusM AS (
--Get Status Metadata
SELECT DISTINCT
EntityName,
QUOTENAME(EntityName + '_Status') AS Alias
FROM dbo.[StatusMetadata]
WHERE LocalizedLabelLanguageCode = #LanguageCode
)
, SQLStatement AS (
--Enumerate all lines in the source table and replace codes with labels where applicable
SELECT CM.EntityName,
--Before the first column of each table, construct a CREATE OR ALTER VIEW statement
CASE WHEN CM.OrdinalPosition = 1
THEN 'CREATE OR ALTER VIEW ' + QUOTENAME(#EnrichedViewSchema) + '.' + CM.EntityName + '
AS
SELECT '
ELSE ' ,'
END
--For each column, check if it needs to be replaced with a suitable localized label
+ CASE
WHEN OSM.OptionSetName IS NOT NULL THEN OSM.Alias + '.[LocalizedLabel] AS ' + REPLACE(QUOTENAME(CM.ColumnName), 'code]', (#EnrichedColumnSuffix COLLATE Latin1_General_100_CI_AS_SC_UTF8) + ']')
WHEN GOSM.OptionSetName IS NOT NULL THEN GOSM.Alias + '.[LocalizedLabel] AS ' + REPLACE(QUOTENAME(CM.ColumnName), 'code]', (#EnrichedColumnSuffix COLLATE Latin1_General_100_CI_AS_SC_UTF8) + ']')
--Uncomment below if you want to show _value columns as well
--WHEN GOSMM.OptionSetName IS NOT NULL THEN '[Base].' + QUOTENAME(CM.ColumnName) + ' as ' + CM.ColumnName + '_value, ' + ((#OptionSetLabelFunctionDatabase COLLATE Latin1_General_100_CI_AS_SC_UTF8) + '.' + (#OptionSetLabelFunctionDatabaseSchema COLLATE Latin1_General_100_CI_AS_SC_UTF8) + '.GetOptionSetLabel(''' + CM.TableName + ''', ''' + CM.ColumnName + ''', '+ '[Base].' + QUOTENAME(CM.ColumnName) + ', ' + #LanguageCode + ') as ' + CM.ColumnName)
--Comment below if you want to show _value columns as well
WHEN GOSMM.OptionSetName IS NOT NULL THEN ((#OptionSetLabelFunctionDatabase COLLATE Latin1_General_100_CI_AS_SC_UTF8) + '.' + (#OptionSetLabelFunctionDatabaseSchema COLLATE Latin1_General_100_CI_AS_SC_UTF8) + '.GetOptionSetLabel(''' + CM.TableName + ''', ''' + CM.ColumnName + ''', '+ '[Base].' + QUOTENAME(CM.ColumnName) + ', ' + #LanguageCode + ') as ' + CM.ColumnName)
WHEN StateM.EntityName IS NOT NULL THEN StateM.Alias + '.[LocalizedLabel] AS ' + REPLACE(QUOTENAME(CM.ColumnName), 'code]', (#EnrichedColumnSuffix COLLATE Latin1_General_100_CI_AS_SC_UTF8) + ']')
WHEN StatusM.EntityName IS NOT NULL THEN StatusM.Alias + '.[LocalizedLabel] AS ' + REPLACE(QUOTENAME(CM.ColumnName), 'code]', (#EnrichedColumnSuffix COLLATE Latin1_General_100_CI_AS_SC_UTF8) + ']')
ELSE '[Base].' + QUOTENAME(CM.ColumnName)
END AS [SQLLine],
CM.OrdinalPosition
FROM CM
LEFT JOIN OSM
ON CM.EntityName = OSM.EntityName
AND CM.ColumnName = OSM.OptionSetName
AND CM.DataType LIKE '%int' --Only include columns with integer data type
LEFT JOIN GOSM
ON CM.ColumnName = GOSM.OptionSetName
AND CM.DataType LIKE '%int' --Only include columns with integer data type
LEFT JOIN GOSMM
ON CM.ColumnName = GOSMM.OptionSetName
AND CM.DataType LIKE '%varchar%' --Only include columns with varchar data type that can potentially have multiselect values
LEFT JOIN StateM
ON CM.EntityName = StateM.EntityName
AND CM.ColumnName = 'statecode'
AND CM.DataType LIKE '%int' --Only include columns with integer data type
LEFT JOIN StatusM
ON CM.EntityName = StatusM.EntityName
AND CM.ColumnName = 'statuscode'
AND CM.DataType LIKE '%int' --Only include columns with integer data type
UNION ALL
--Construct the first line of the FROM clause, referencing external tables created by Synapse Link for Dataverse
SELECT DISTINCT
CM.EntityName,
'FROM ' + QUOTENAME(DB_NAME()) + '.' + QUOTENAME(CM.TableSchema) + '.' + QUOTENAME(CM.TableName) + ' AS Base' AS SQLLine,
10000 AS OrdinalPosition
FROM CM
UNION ALL
--Construct LEFT JOIN statements for each relevant OptionSetMetadata field
SELECT DISTINCT OSM.EntityName AS EntityName,
' LEFT JOIN ' + QUOTENAME(DB_NAME()) + '.[dbo].[OptionSetMetadata] AS ' + OSM.Alias + '
ON ' + OSM.Alias + '.EntityName = ''' + OSM.EntityName + '''
AND ' + OSM.Alias + '.OptionSetName = ''' + OSM.OptionSetName + '''
AND [Base].' + QUOTENAME(OSM.OptionSetName) + ' = ' + OSM.Alias + '.[Option]
AND ' + OSM.Alias + '.LocalizedLabelLanguageCode = ' + #LanguageCode + '' AS SQLLine,
20000 AS OrdinalPosition
FROM OSM
JOIN CM
ON CM.EntityName = OSM.EntityName
AND CM.ColumnName = OSM.OptionSetName
WHERE CM.DataType LIKE '%int' --Only capture columns with Integer Data Types
UNION ALL
--Construct LEFT JOIN statements for each relevant GlobalOptionSetMetadata field
SELECT DISTINCT CM.EntityName AS EntityName,
' LEFT JOIN ' + QUOTENAME(DB_NAME()) + '.[dbo].[GlobalOptionSetMetadata] AS ' + Alias + '
ON ' + Alias + '.OptionSetName = ''' + OptionSetName + '''
AND [Base].' + QUOTENAME(OptionSetName) + ' = ' + Alias + '.[Option]
AND ' + Alias + '.LocalizedLabelLanguageCode = ' + #LanguageCode + '' AS SQLLine,
30000 AS OrdinalPosition
FROM GOSM
JOIN CM
ON CM.ColumnName = GOSM.OptionSetName
WHERE CM.DataType LIKE '%int' --Only capture columns with Integer Data Types
UNION ALL
--Construct LEFT JOIN statements for each relevant State Metadata field
SELECT DISTINCT CM.EntityName AS EntityName,
' LEFT JOIN ' + QUOTENAME(DB_NAME()) + '.[dbo].[StateMetadata] AS ' + StateM.Alias + '
ON ' + StateM.Alias + '.EntityName = ''' + StateM.EntityName + '''
AND [Base].statecode' + ' = ' + StateM.Alias + '.[State]
AND ' + StateM.Alias + '.LocalizedLabelLanguageCode = ' + #LanguageCode + '' AS SQLLine,
40000 AS OrdinalPosition
FROM StateM
JOIN CM
ON CM.EntityName = StateM.EntityName
AND CM.ColumnName = 'statecode'
WHERE CM.DataType LIKE '%int' --Only capture columns with Integer Data Types
UNION ALL
--Construct LEFT JOIN statements for each relevant Status Metadata field
SELECT DISTINCT CM.EntityName AS EntityName,
' LEFT JOIN ' + QUOTENAME(DB_NAME()) + '.[dbo].[StatusMetadata] AS ' + StatusM.Alias + '
ON ' + StatusM.Alias + '.EntityName = ''' + StatusM.EntityName + '''
AND [Base].statuscode' + ' = ' + StatusM.Alias + '.[Status]
AND ' + StatusM.Alias + '.LocalizedLabelLanguageCode = ' + #LanguageCode + '' AS SQLLine,
40000 AS OrdinalPosition
FROM StatusM
JOIN CM
ON CM.EntityName = StatusM.EntityName
AND CM.ColumnName = 'statuscode'
WHERE CM.DataType LIKE '%int' --Only capture columns with Integer Data Types
UNION ALL
--Add statement terminator
SELECT DISTINCT
EntityName,
'; ' + CHAR(10) AS SQLLine,
100000 AS OrdinalPosition
FROM CM
)
--Construct individual statements to create views (1 view per row)
--Since CREATE VIEW statement must be the first statement in a batch, assign each view definition to a variable
--and use the EXEC(#variable) command to create view as part of its own, separate batch.
, ViewDefinitions AS (
SELECT 'DECLARE #' + EntityName + ' NVARCHAR(MAX) = ''
' + REPLACE(STRING_AGG(CAST(SQLLine as varchar(MAX)), CHAR(10)) WITHIN GROUP (ORDER BY EntityName, OrdinalPosition, SQLLine), '''', '''''') + ''' ' + CHAR(10) + 'EXEC [' + (#EnrichedViewDatabase COLLATE Latin1_General_100_CI_AS_SC_UTF8) + '].dbo.sp_executesql #' + EntityName + CHAR(10) AS ViewDefinition
FROM SQLStatement
GROUP BY EntityName
)
--Construct a comprehensive SQL statement to create all views
SELECT #SQL = STRING_AGG(ViewDefinition, ';' + CHAR(10) + CHAR(10))
FROM ViewDefinitions
--Return a preview of the SQL Script to be generated or go ahead and create the views.
IF #PreviewOnly = 1
BEGIN
--Return the final SQL statement
SELECT '--================================================================================================='+ CHAR(10) +' ' + CHAR(10) AS [--SQL Statement]
UNION ALL
SELECT '-- A preview of the script to generate enriched views is provided below.' AS [--SQL Statement]
UNION ALL
SELECT '-- No database objects have been created.' AS [--SQL Statement]
UNION ALL
SELECT '-- Re-run this script with the #PreviewOnly parameter set to 0 to actually create the views.' AS [--SQL Statement]
UNION ALL
SELECT '--================================================================================================='+ CHAR(10) +' ' + CHAR(10) AS [--SQL Statement]
UNION ALL
SELECT VALUE AS [--SQL Statement] FROM STRING_SPLIT((#SQL COLLATE Latin1_General_100_CI_AS_SC_UTF8), CHAR(10))
END
ELSE
BEGIN
--Execute the SQL statement
PRINT 'Beginning views creation'
--PRINT #SQL
EXEC sp_executesql #SQL
PRINT 'Completed views creation'
END
Operation DROP FUNCTION is not allowed for a replicated database
In Synapse Serverless SQL there are some operations are not allowed
You may see above problems while attempting to create SQL objects, users, or alter permissions in a database. When you attempt to create objects in a database that is shared with the Spark pool, this error is given. Read-only databases are replicated from Apache Spark pools. A replicated database we cannot create new objects added using T-SQL.
To workaround this, create a custom SQL database (serverless) that will contain the custom schemas, views, and functions that will reference Lake database external tables using the 3-part names.
Refer - Operation isn't allowed for a replicated database
I want to update cells in an SQL table depending on the type of its column. More specifically, I want to replace NULL values by either '',0,0.0 or '1900-01-01' depending on the column type. The table has 60 columns and so a manual approach is awkward.
The approach I have tried uses metadata and I want to generate an update script.
Use case is for an import into Excel where the NULL values must be taken care off in a column type dependant way.
As discussed in the comments, the easiest way seems to be use ''. The only exception (I found) was for decimal/numeric, where implicit and explicit conversion is not allowed.
As shown below, you can write the statement out (but I exclude the decimal column) or do it dynamically:
USE Sandbox;
GO
CREATE TABLE TestTable (ID int IDENTITY(1,1),
SomeInt int,
SomeDecimal decimal(10,2),
SomeFloat float,
SomeMoney money,
SomeDate date,
SomeDatetime datetime,
SomeTime time,
SomeDatetime2 datetime2(1),
SomeVarbinary varbinary(12),
SomeVarchar varchar(10),
SomeBit bit);
INSERT INTO dbo.TestTable (SomeInt,
SomeDecimal,
SomeFloat,
SomeMoney,
SomeDate,
SomeDatetime,
SomeTime,
SomeDatetime2,
SomeVarbinary,
SomeVarchar,
SomeBit)
VALUES(NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL),
(NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL);
GO
UPDATE dbo.TestTable
SET SomeInt = ISNULL(SomeInt,''),
--SomeDecimal = ISNULL(SomeDecimal,''),
SomeFloat = ISNULL(SomeFloat,''),
SomeMoney = ISNULL(SomeMoney,''),
SomeDate = ISNULL(SomeDate,''),
SomeDatetime = ISNULL(SomeDatetime,''),
SomeTime = ISNULL(SomeTime,''),
SomeDatetime2 = ISNULL(SomeDatetime2,''),
SomeVarbinary = ISNULL(SomeVarbinary,CONVERT(varbinary,'')),
SomeVarchar = ISNULL(SomeVarchar,''),
SomeBit = ISNULL(SomeBit,'')
WHERE ID = 1;
GO
DECLARE #Schema sysname = N'dbo',
#Table sysname = N'TestTable';
DECLARE #SQL nvarchar(MAX);
SET #SQL = N'UPDATE ' + QUOTENAME(#Schema) + N'.' + QUOTENAME(#Table) + NCHAR(13) + NCHAR(10) +
N'SET ' + STUFF((SELECT N',' + NCHAR(13) + NCHAR(10) +
N' ' + QUOTENAME(C.COLUMN_NAME) + N' = ISNULL(' + QUOTENAME(C.COLUMN_NAME) + N',' + CASE C.DATA_TYPE WHEN N'decimal' THEN '0)'
WHEN N'numeric' THEN '0)'
ELSE N'CONVERT(' + QUOTENAME(C.DATA_TYPE) + N',N''''))'
END
FROM INFORMATION_SCHEMA.COLUMNS C
WHERE C.TABLE_SCHEMA = #Schema
AND C.TABLE_NAME = #Table
AND C.COLUMN_NAME != 'ID' --You'll need a better method to ignore your ID column
FOR XML PATH(N''),TYPE).value('.','nvarchar(MAX)'),1,7,N'') + NCHAR(13) + NCHAR(10) +
N'WHERE ID = 2;'; --you won't want this if you're doing every row.
PRINT #SQL;
EXEC sp_executesql #SQL;
GO
SELECT *
FROM dbo.TestTable;
GO
DROP TABLE dbo.TestTable
db<>fiddle
If you're using SQL Server 2017+, you can simplify the above by using STRING_AGG instead of FOR XML PATH and STUFF to create a "delimited" list of the columns in the table.
SELECT COLUMN_NAME AS colname, DATA_TYPE AS coltype
INTO #result_columns
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'mytable';
WITH cte(colname, default_value) AS (
SELECT colname,
CASE
WHEN coltype IN ('varchar','nvarchar') THEN ''''''
WHEN coltype IN ('int','money') THEN '0'
WHEN coltype IN ('date') THEN '''1900-01-01'''
WHEN coltype IN ('float') THEN '0.0'
END AS default_value
FROM #result_columns
)
SELECT 'UPDATE mytable SET ' + colname + '=' + default_value + ' WHERE ' + colname + ' IS NULL;'
FROM cte;
-- result :
UPDATE mytable SET Caracteristique='' WHERE Caracteristique IS NULL;
UPDATE mytable SET Couleur='' WHERE Couleur IS NULL;
UPDATE mytable SET Forme='' WHERE Forme IS NULL;
UPDATE mytable SET Machine='' WHERE Machine IS NULL;
UPDATE mytable SET QteChargee=0.0 WHERE QteChargee IS NULL;
I have a table which has more than 30 columns(all are varchar). I need to list out all the columns which contains blank i.e.' ' values.
I tried using 'coalesce' but it is only for NULL.
The following query will give you all the columns in a table that might have null or '' values.
It is written so that you can run it for all tables in your database but you can limit it to a single table, as I have done for this specific example, checking a table called testingNulls:
--two variables needed for table name and column name, when looping through all tables
declare #table varchar(255), #col varchar(255), #sql varchar(max)
--this will be used to store the result, to have one result set instead of one row per each cursor cycle
if object_id('tempdb..#nullcolumns') is not null drop table #nullcolumns
create table #nullcolumns (tablename varchar(255), columnname varchar(255))
declare getinfo cursor for
select t.name tablename, c.name
from sys.tables t join sys.columns c on t.object_id = c.object_id
where t.name = 'testingnulls' --here the condition for the table name
open getinfo
fetch next from getinfo into #table, #col
while ##fetch_status = 0
begin
select #sql = 'if exists (select top 1 * from [' + #table + '] where [' + #col + '] is null or [' + #col + '] like '''' ) begin insert into #nullcolumns select ''' + #table + ''' as tablename, ''' + #col + ''' as all_nulls end'
print(#sql)
exec(#sql)
fetch next from getinfo into #table, #col
end
close getinfo
deallocate getinfo
--this should be the result you need:
select * from #nullcolumns
You can see a working example here. I hope this is what you need.
List all columns that contain a blank in some record? You'd use a query per column and collect the results with UNION ALL:
select 'COL1' where exists (select * from mytable where col1 like '% %')
union all
select 'COL2' where exists (select * from mytable where col2 like '% %')
union all
...
union all
select 'COL30' where exists (select * from mytable where col30 like '% %');
If you want like select * from [your_table_name] where [col1] = '' and [col2] = ''....., then use dynamic sql query like below.
Query
declare #sql as varchar(max);
select #sql = 'select * from [your_table_name] where '
+ stuff((
select ' and [' + [column_name] + '] = ' + char(39) + char(39)
from information_schema.columns
where table_name = 'your_table_name'
for xml path('')
)
, 1, 5, ''
);
exec(#sql);
Update
Or else if you want to list the column names which have a blank value, then you can use the below dynamic sql query.
Query
declare #sql as varchar(max);
select #sql = stuff((
select ' union all select ' + [column_name] + ' as [col1], '
+ char(39) + [column_name] + char(39) + ' as [col2]'
+ ' from your_table_name'
from information_schema.columns
where table_name = 'your_table_name'
for xml path('')
)
, 1, 11, ''
);
set #sql = 'select distinct t.col2 as [blank_cols] from(' + #sql
+ ')t
where coalesce(ltrim(rtrim(t.col1)), ' + char(39) + char(39) + ') = '
+ char(39) + char(39) + ';';
exec(#sql);
Find a demo here
But still I'm not sure that this is what you are looking out for.
you have not many choices but to specify all the columns in your where clause
WHERE COL1 = '' AND COL2 = '' AND COL3 = '' AND . . .
or you can use Dynamic SQL to form your query, but that is not an easy path to go
If you want to count number of columns having '' value in a table (not for each row) then use the following
SELECT max(CASE WHEN col1 = '' THEN 1 ELSE 0 END) +
max(CASE WHEN col2 = '' THEN 1 ELSE 0 END) +
max(CASE WHEN col3 = '' THEN 1 ELSE 0 END) +
...
FROM t
demo
I created a dynamic SQL script that you can use by providing the table name only
Here it is
declare #sql nvarchar(max)
declare #table sysname = 'ProductAttributes'
select #sql =
'select * from ' + #table + ' where ' +
string_agg('[' + name + '] = '' '' ', ' and ')
from sys.columns
where object_id = OBJECT_ID(#table)
select #sql
exec sp_executesql #sql
Unfortunately, for SQL string concatenation String_Agg function is new with SQL Server 2017
But it is also possible to use SQL XML Path to concatenate WHERE clause fragments
SELECT #sql = 'select * from ' + #table + ' where ' +
STUFF(
(
SELECT
' and ' + '[' + [name] + '] = '' '' '
from sys.columns
where object_id = OBJECT_ID(#table)
FOR XML PATH(''),TYPE
).value('.','VARCHAR(MAX)'
), 1, 5, ''
)
select #sql as sqlscript
exec sp_executesql #sql
So simply I'm doing something similar to:
select
[BadData], [WorseDate], [IQuitData]
into
#BadDataTempTable
from
SomeoneElsesMess
what I want to do now is something similar to:
Select
Replace(#BadDataTempTable.*, ',', ' ')
from
#BadDataTempTable -- Replace all commas in every column `with a space--`
Is this possible? If so please show me the easiest (non-function) way to do so.
Thanks. SQL Server 2012 I think. I'm using SSMS 17
No, the columns have to be specified. You could use dynamic SQL to build your update / query. Then just copy the command you want from the results.
Maybe this will help get you started:
BEGIN
-- Set the replace value
DECLARE #ls_replaceValue NVARCHAR(MAX) = ',';
-- Set the with value
DECLARE #ls_withValue NVARCHAR(MAX) = ' ';
-- Set the table name we want to query
DECLARE #ls_table NVARCHAR(MAX) = 'some_table';
-- Get all of the columns and provide the replace parameters
DECLARE #ls_columns NVARCHAR(MAX) = '';
SELECT #ls_columns = #ls_columns + ', ' + name + ' = REPLACE(' + name + ', ' + '' + '''' + REPLACE(#ls_replaceValue, '''', '''''''') + '''' + ', ' + '''' + REPLACE(#ls_withValue, '''', '''''''') + '''' + ')'
FROM sys.all_columns
WHERE object_id = OBJECT_ID(#ls_table)
AND collation_name IS NOT NULL; -- Skip columns that aren't character based
-- Remove the first ', ' from the column list
SET #ls_columns = SUBSTRING(#ls_columns, 3, LEN(#ls_columns));
IF #ls_columns = ''
BEGIN
PRINT 'Table not found'
RETURN
END
-- Build a query
DECLARE #ls_query_sql NVARCHAR(MAX) = '';
SET #ls_query_sql = 'SELECT ' + #ls_columns + ' FROM ' + #ls_table;
-- Show the results
SELECT #ls_query_sql AS querySQL;
END
Just since the OP asked about how you might do this in dynamic SQL, here's how I'd approach it. Basically get the table schema information and concatenate all the columns, plus the REPLACE logic you want using FOR XML. This basically constructs the statement Rigerta posted, but does it dynamically.
use tempdb
go
if object_id('tempdb.dbo.#SomeoneElsesBadData') is not null drop table #SomeoneElsesBadData
create table #SomeoneElsesBadData
(
BadData varchar(250),
WorseData varchar(250),
IQuitData varchar(250)
)
declare #sql nvarchar(max)
select #sql = 'select '
+ stuff((select ', '
+ name
+ ' = replace(' + name + ''','', '''')'
from tempdb.sys.columns
where object_id = object_id('tempdb.dbo.#SomeoneElsesBadData')
for xml path('')), 1, 1, '')
+ ' into #BadDataTempTable
from #SomeoneElsesBadData'
exec sp_executesql #sql
All things being equal, the data should probably be cleaned before it gets into SQL, but reality is rarely fair.
This is probably a bit of a limited, but valuable scenario. I have a SQL Server 2008 database with a table that has millions of records. There appears to be an intermittent problem with several of the records. I'm trying to repro the problem. In an effort to do this, I finally got the ID of an offending record. I would like to generate an INSERT statement associated with this single record in my PROD database. Then I can easily migrate it into my TESTING database in an effort to repro and resolve the problem.
Basically, I need to generate a single INSERT statement for a single record from a single table where I know the primary key value of the record.
Does anyone have any ideas of how I can accomplish this? Essentially, I want to generate insert statements on a conditional basis.
Thank you!
First try to recreate what you want to insert with a SELECT statement.
After that you can insert into the table with a INSERT INTO like this:
INSERT INTO tablename
SELECT ....
If they are on different servers, you can use INSERT like this:
INSERT INTO tablename VALUES (...)
using the values given by the SELECT in the other server fill the values in the insert.
In your specific case I think you can do this:
CREATE PROCEDURE dbo.GenerateSingleInsert
#table NVARCHAR(511), -- expects schema.table notation
#pk_column SYSNAME, -- column that is primary key
#pk_value INT -- change data type accordingly
AS
BEGIN
SET NOCOUNT ON;
DECLARE #cols NVARCHAR(MAX), #vals NVARCHAR(MAX),
#valOut NVARCHAR(MAX), #valSQL NVARCHAR(MAX);
SELECT #cols = N'', #vals = N'';
SELECT #cols = #cols + ',' + QUOTENAME(name),
#vals = #vals + ' + ' + REPLICATE(CHAR(39),3) + ','
+ REPLICATE(CHAR(39),3) + ' + ' + REPLICATE(CHAR(39),2) + '+'
+ 'RTRIM(' + CASE WHEN system_type_id IN (40,41,42,43,58,61) THEN
'CONVERT(CHAR(8), ' + QUOTENAME(name) + ', 112) + '' ''
+ CONVERT(CHAR(14), ' + QUOTENAME(name) + ', 14)'
ELSE 'REPLACE(' + QUOTENAME(name) + ','''''''','''''''''''')' END + ')
+ ' + REPLICATE(CHAR(39),2)
FROM sys.columns WHERE [object_id] = OBJECT_ID(#table)
AND system_type_id <> 189 -- can't insert rowversion
AND is_computed = 0; -- can't insert computed columns
SELECT #cols = STUFF(#cols, 1, 1, ''),
#vals = REPLICATE(CHAR(39), 4) + ' + ' + STUFF(#vals, 1, 13, '')
+ REPLICATE(CHAR(39), 2);
SELECT #valSQL = N'SELECT #valOut = ' + #vals + ' FROM ' + #table + ' WHERE '
+ QUOTENAME(#pk_column) + ' = ''' + RTRIM(#pk_value) + ''';';
EXEC sp_executesql #valSQL, N'#valOut NVARCHAR(MAX) OUTPUT', #valOut OUTPUT;
SELECT SQL = 'INSERT ' + #table + '(' + #cols + ') SELECT ' + #valOut;
END
GO
So let's try it out:
CREATE TABLE dbo.splunge
(
ID INT, dt DATETIME, rv ROWVERSION, t NVARCHAR(MAX)
);
INSERT dbo.splunge(ID, dt, t)
SELECT 1, GETDATE(), 'foo'
UNION ALL SELECT 2, GETDATE(), 'bar'
UNION ALL SELECT 3, GETDATE(), 'O''Brien';
EXEC dbo.GenerateSingleInsert N'dbo.splunge', N'ID', 1;
SQL
-------------
INSERT dbo.splunge([ID],[dt],[t]) SELECT '1','20120517 10:07:07:330','foo'
EXEC dbo.GenerateSingleInsert N'dbo.splunge', N'ID', 2;
SQL
-------------
INSERT dbo.splunge([ID],[dt],[t]) SELECT '2','20120517 10:07:07:330','bar'
EXEC dbo.GenerateSingleInsert N'dbo.splunge', N'ID', 3;
SQL
-------------
INSERT dbo.splunge([ID],[dt],[t]) SELECT '3','20120517 10:07:07:330','O''Brien'
If there is an IDENTITY column you may need to set SET IDENTITY_INSERT ON for the TEST table, and verify that there is no collision. Probably about 500 caveats I should mention, I haven't tested all data types, etc.
However in the more general case there is a lot more to it than this. Vyas K has a pretty robust stored procedure that should demonstrate how complicated it can get:
http://vyaskn.tripod.com/code/generate_inserts_2005.txt
You are probably far better off using a tool like Red-Gate's SQL Data Compare to pick a specific row and generate an insert for you. As I've blogged about, paying for a tool is not just about the money, it's about the hours of troubleshooting and bug-fixing that someone else has already done for you.
Aaron,
I liked your code, it solved a problem for me. I ran into a few issues using it (like you said I would) with nulls and the text type so I made some changes to address those issues.
ALTER PROCEDURE dbo.GenerateSingleInsert
#table NVARCHAR(511), -- expects schema.table notation
#pk_column SYSNAME, -- column that is primary key
#pk_value INT -- change data type accordingly
AS
BEGIN
SET NOCOUNT ON;
DECLARE #cols NVARCHAR(MAX), #vals NVARCHAR(MAX),
#valOut NVARCHAR(MAX), #valSQL NVARCHAR(MAX);
SELECT #cols = N'', #vals = N'';
SELECT #cols = #cols + ',' + QUOTENAME(name),
#vals = #vals + ' + '','' + ' + 'ISNULL('+REPLICATE(CHAR(39),4)+'+RTRIM(' +
CASE WHEN system_type_id IN (40,41,42,43,58,61) -- datetime types
THEN
'CONVERT(CHAR(8), ' + QUOTENAME(name) + ', 112) + '' ''+ CONVERT(CHAR(14), ' + QUOTENAME(name) + ', 14)'
WHEN system_type_id IN (35) -- text type NOTE: can overflow
THEN
'REPLACE(CAST(' + QUOTENAME(name) + 'as nvarchar(MAX)),'+REPLICATE(CHAR(39),4)+','+REPLICATE(CHAR(39),6)+')'
ELSE
'REPLACE(' + QUOTENAME(name) + ','+REPLICATE(CHAR(39),4)+','+REPLICATE(CHAR(39),6)+')'
END
+ ')+' + REPLICATE(CHAR(39),4) + ',''null'') + '
FROM sys.columns WHERE [object_id] = OBJECT_ID(#table)
AND system_type_id <> 189 -- can't insert rowversion
AND is_computed = 0; -- can't insert computed columns
SELECT #cols = STUFF(#cols, 1, 1, ''),
#vals = REPLICATE(CHAR(39),2) + STUFF(#vals, 1, 6, '') + REPLICATE(CHAR(39),2) ;
SELECT #valSQL = N'SELECT #valOut = ' + #vals + ' FROM ' + #table + ' WHERE '
+ QUOTENAME(#pk_column) + ' = ''' + RTRIM(#pk_value) + ''';';
EXEC sp_executesql #valSQL, N'#valOut NVARCHAR(MAX) OUTPUT', #valOut OUTPUT;
SELECT SQL = 'INSERT ' + #table + '(' + #cols + ') SELECT ' + #valOut;
END