Mask values in a SQL Query string for SQL Server - sql

I am a SQL Server DBA. I would like to write a procedure which I can provide to rest of my team where they can view the text for currently running queries on the server (Similar to how we view in sp_who2) but with all the values masked.
Examples:
Query text
Query text after Masking
Select * from sometable where rating = '4'
Select * from sometable where rating = '****'
Select name, id from sometable where id = '3233'
Select name, id from sometable where id = '****'
UPDATE Customers SET ContactName = 'Alfred Schmidt' WHERE CustomerID = 1;
UPDATE Customers SET ContactName = '****' WHERE CustomerID = ****;
INSERT INTO Customers (CustomerName, ContactName) VALUES ('Cardinal', 'Tom B. Erichsen');
INSERT INTO Customers (CustomerName, ContactName) VALUES ('*****', '****');

If I understand correctly your issue.
You can use this query:
select
r.session_id,
r.status,
r.command,
r.cpu_time,
r.total_elapsed_time,
t.text
from sys.dm_exec_requests as r
cross apply sys.dm_exec_sql_text(r.sql_handle) as t
e.g.
I run it on my SQL server right now:
(#P1 nvarchar(5),#P2 bigint,#P3 int,#P4 numeric(28, 12),#P5 nvarchar(5),#P6 datetime,#P7 datetime)
SELECT SUM(A.SETTLEAMOUNTCUR) FROM CUSTSETTLEMENT A,CUSTTRANS B WHERE ((A.DATAAREAID=#P1) AND (((A.TRANSRECID=#P2) AND (A.CANBEREVERSED=#P3)) AND (A.SETTLEAMOUNTCUR<>#P4))) AND ((B.DATAAREAID=#P5) AND (((B.RECID=A.OFFSETRECID) AND (B.TRANSDATE>=#P6)) AND (B.TRANSDATE<=#P7)))
All variables are hidden.

You could try some XML-trickery to handle the strings.
First replace all single quotes with an empty tag <X/> to get a XML that looks like this.
INSERT INTO Customers (CustomerName, ContactName)
VALUES (<X />Cardinal<X />, <X />Tom B. Erichsen<X />);
Then you shred the xml to get the text nodes and the node numbers where mod 2 is 0 is the ones you want to mask.
After that you can rebuild your query string using the mask values.
I have not found a way to deal with numbers other then removing all numbers from the query using Translate or nested replace and that will of course also remove numbers from table names and column names as well.
You could try something like this.
declare #S nvarchar(max);
declare #X xml;
set #S = N'UPDATE Customers SET ContactName = ''Alfred Schmidt'' WHERE CustomerID = 1;';
set #X = replace(#S, '''', '<X/>');
with C as
(
select T.X.value('.', 'nvarchar(max)') as V,
row_number() over(order by T.X) as RN
from #X.nodes('text()') as T(X)
)
select #S = (
select case when C.RN % 2 = 0 then '''*****''' else C.V end
from C
order by C.RN
for xml path(''), type
).value('text()[1]', 'nvarchar(max)');
set #S = translate(#S, '0123456789', '**********')
print #S;
Result:
UPDATE Customers SET ContactName = '*****' WHERE CustomerID = *;
Note: Just realized that this solution does not handle the cases where the string values contains single quotes but I think this is something that possibly can inspire more robust solution so I will leave it here.

sys.sp_get_query_template
fiddle
declare #t nvarchar(max), #p nvarchar(max);
declare #q nvarchar(max) = 'UPDATE Customers SET ContactName = N''Alfred Schmidt'' WHERE CustomerID = 1 AND Rate = 0.75 AND Rver = 0x0102 AND DateCreated = dateadd(day, -10, ''202z0818'')';
exec sys.sp_get_query_template #querytext = #q, #templatetext = #t OUTPUT, #parameters = #p OUTPUT;
select p,
case when tp like '%int' then cast('****' as nvarchar(40))
when tp like 'decimal(%' or tp like 'numeric(%' then '**.**'
when tp like '%binary(%' then '0x****'
when tp like 'n%char%' then 'N''****'''
else '''****'''
end as rv
into #t
from
(
select *, '#'+left(s.value, charindex(' ', s.value+' ')-1) as p, stuff(s.value, 1, charindex(' ', s.value), '') as tp
from string_split(replace(#p, ',#', '#'), '#') as s
where s.value <> ''
) as ss;
update #t
set #t = replace(#t, p, rv);
select #q union all select #t;

Related

Transform a SELECT * query to string

I have a query that returns a row
SELECT *
FROM table
WHERE id = 1;
I want to save the result into a nvarchar sql variable. I have seen similar questions Convert SQL Server result set into string but they only use select with the name of the columns, never with *.
select *
from table
where id = 1
for xml path ('')
However the answer is <column1>value1</column1> <column2>value2</column2> and I just want it to be value1, value2
Is there a way to achieve this? thank you!
If open to a helper function.
This will convert virtually any row, table or query to a string (delimited or not).
In the following examples I selected a PIPE delimiter with a CRLF line terminator.
Please note the usage and placement of _RN when a line terminator is required. Also note the ,ELEMENTS XSINIL ... this will included null values as empty string. If you want to exclude null values, simply omit the ,ELEMENTS XSINIL
Example as Entire Table or dbFiddle
Declare #YourTable Table (id int,[col_1] varchar(50),[col_2] varchar(50),[col_3] varchar(50),[col_n] varchar(50)) Insert Into #YourTable Values
(1,'data1','data2','data3','data4')
,(2,'data5','data6','data7','data8')
-- Entire Table
Declare #XML xml = (Select *,_RN=Row_Number() over (Order By (Select null)) From #YourTable for XML RAW,ELEMENTS XSINIL )
Select [dbo].[svf-str-Data-To-Delimited]('|',char(13)+char(10),#XML)
Returns
1|data1|data2|data3|data4
2|data5|data6|data7|data8
Example as Row Based
Select A.ID
,AsAString = [dbo].[svf-str-Data-To-Delimited]('|',char(13)+char(10),B.XMLData)
From #YourTable A
Cross Apply ( values ( (select a.* for xml RAW,ELEMENTS XSINIL )) )B(XMLData)
Returns
ID AsAString
1 1|data1|data2|data3|data4
2 2|data5|data6|data7|data8
The Function if Interested
CREATE Function [dbo].[svf-str-Data-To-Delimited] (#Delim varchar(50),#EOL varchar(50),#XML xml)
Returns varchar(max)
Begin
Return(
Select convert(nvarchar(max),(
Select case when Item='_RN' then ''
else case when nullif(lead(Item,1) over (Order by Seq),'_RN') is not null
then concat(Value,#Delim)
else concat(Value,#EOL)
end
end
From (
Select Seq = row_number() over(order by (select null))
,Item = xAttr.value('local-name(.)', 'nvarchar(100)')
,Value = xAttr.value('.','nvarchar(max)')
From #XML.nodes('/row/*') xNode(xAttr)
) A
Order By Seq
For XML Path (''),TYPE).value('.', 'nvarchar(max)') )
)
End
You can easily store the result as an XML string:
select *
from (values (1, 'x', getdate())) v(id, a, b)
where id = 1
for xml path ('');
Or as a JSON string:
select *
from (values (1, 'x', getdate())) v(id, a, b)
where id = 1
for json auto;
If you don't mind Using dynamic SQL (and INFORMATION_SCHEMA dictionary), for example, for SQL Server this works:
DECLARE #sql nvarchar(max) = '',
#result nvarchar(max),
#id int = 1
SELECT #sql += '+'',''+convert(nvarchar,' + QUOTENAME(column_name) +')' from INFORMATION_SCHEMA.columns where table_name = 'Student'
SET #sql = 'select #result=' + stuff(#sql,1,5,'') + ' from student where id = ' + CAST(#id as nvarchar)
EXECUTE sp_executesql #sql, N'#result nvarchar(max) OUTPUT', #result=#result OUTPUT
SELECT #result as MyOutput

How to put a comma separated value from a column in a table into SQL IN operator?

I have a table which has a column in which I am storing a comma separated text with single quotes for each of the comma separated values. These values are employee IDs. This is how it looks
Now, I have a SQL query wherein I need to put the value from this column into a SQL IN operator. Something like this:
select *
from EMPLOYEE_MASTER
where EMPLOYEEID IN (select CM_CONFIG_VALUE
from ADL_CONFIG_MAST_T
where CM_CONFIG_KEY like 'ATT_BIOMETRIC_OU_ID'
)
But this, does not work, the query when executed returns 0 rows whereas if I execute the query normally like below, it works.
select *
from EMPLOYEE_MASTER
where EMPLOYEEID IN('9F3DD4B791554DDE','C9B90D62851D43AB','828CB9E6204B4DDC')
Please suggest what I should do here. I have tried using substring to remove the first and the last character as well assuming that single quotes might be the issue, but that does not work either.
select * from EMPLOYEE_MASTER where EMPLOYEEID IN(select EMPLOYEEID from ADL_CONFIG_MAST_T where CM_CONFIG_KEY like 'ATT_BIOMETRIC_OU_ID')
column should be same in where COLUMNNAME IN (select COLUMNNMAE from tablename)
You can create a temp varible and then use exec command to get the desired result.
declare #temp varchar(200)
select #temp=CM_CONFIG_VALUE
from ADL_CONFIG_MAST_T
where CM_CONFIG_KEY like 'ATT_BIOMETRIC_OU_ID'
exec('select *
from EMPLOYEE_MASTER
where EMPLOYEEID IN (' + #temp + ')')
Try This:
DECLARE #ID VARCHAR(500);
DECLARE #Number VARCHAR(500);
DECLARE #comma CHAR;
SET #comma = ','
SET #ID = (select CM_CONFIG_VALUE
from ADL_CONFIG_MAST_T
where CM_CONFIG_KEY like %ATT_BIOMETRIC_OU_ID% + #comma);
Create table #temp (EMPLOYEEID varchar(500))
WHILE CHARINDEX(#comma, #ID) > 0
BEGIN
SET #Number = SUBSTRING(#ID, 0, CHARINDEX(#comma, #ID))
SET #ID = SUBSTRING(#ID, CHARINDEX(#comma, #ID) + 1, LEN(#ID))
Insert into #temp
select #Number
END
select *
from EMPLOYEE_MASTER
where EMPLOYEEID IN(select EMPLOYEEID from #temp)
The reason you are not getting it in your query is because your inner query returns only one row. So your query searches for '9F3DD4B791554DDE','C9B90D62851D43AB','828CB9E6204B4DDC' as as single record.
If your compatibility level is greater than or equal to 130 you can use STRING_SPLIT() function. Then your query would be
SELECT *
FROM EMPLOYEE_MASTER
WHERE EMPLOYEEID IN
(SELECT value AS empid
FROM ADL_CONFIG_MAST_T CROSS APPLY string_split(CM_CONFIG_VALUE, ',' )
WHERE CM_CONFIG_KEY LIKE 'ATT_BIOMETRIC_OU_ID' )
What this actually does is, it splits the CM_CONFIG_VALUE with ',' and returns them as rows. This is the value column I have referred. Then you use them with the IN clause.
Hope this helps!
Direct IN condition will not work here. You have split your string before searching. You can do that with XML options in SQL SERVER 2014
SELECT *
FROM EMP
WHERE EMPID IN (
SELECT a.c.value('.', 'VARCHAR(1000)')
FROM (
SELECT x = CAST('<a>' +
REPLACE(REPLACE(CM_CONFIG_VALUE , ',', '</a><a>'),'''','') + '</a>' AS XML )
FROM ADL_CONFIG_MAST_T
-- WHERE <your_condition>
) m
CROSS APPLY x.nodes('/a') a(c))
CHECK DEMO HERE
For the version 2016 and above you can use STRING_SPLIT with Compatibility level 130

query not retrieving values returned from split string

I generate comma seperated string and add single quite to each numbers
Here is how i do it
DECLARE #IDs NVARCHAR(max)
SELECT #IDs = COALESCE(#IDs +',', '') + ''''
+ Cast([mynos] AS NVARCHAR(255)) + ''''
FROM mytable
WHERE id = 22
If i print variable #IDs then i get below output
'78888','3333','1222'
When i use same variable in this query then query doesnt return any value
SELECT *
FROM table1
WHERE ids IN ( #IDs )
How to fix this?
It doesn't work as your query is effectively doing this:
SELECT *
FROM TABLE
WHERE Ids IN ('''78888'',''3333',''1222''');
Which would also be equivalent to:
SELECT *
FROM TABLE
WHERE Ids = '''78888'',''3333',''1222''';
If you want to do the query as you have done, you'll need to split your delomited data out again. As you're using SQL Server 2012, you can't make use of STRING_SPLIT, so you'll need to a different one; such as Jeff Moden's DelimitedSplit8K. Then you can do:
SELECT *
FROM TABLE
WHERE IDs IN (SELECT items
FROM dbo.DelimitedSplit8K (#IDs,','));
However, why are you not simply doing...
SELECT *
FROM TABLE T
WHERE EXISTS (SELECT 1
FROM myTable mT
WHERE mT.Id = 22
AND mT.myNos = T.Ids);
You can use dynamic query #id is string variable not multi-value argument
DECLARE #IDs nVARCHAR(MAX)
SELECT #IDs = COALESCE(#IDs +',' ,'') + '''' + CAST([myNos] AS nVARCHAR(255)) + ''''
FROM myTable WHERE Id = 22
DECLARE #query nVARCHAR(MAX)
SET #query = "Select * from table1 where Ids in ("+#IDs+")"
EXECUTE sp_executesql #query
I tried below and it worked
select * from table1 where id in (select mynos from mytable where id = 22)
Thanks to #Larnu for giving me idea

T-SQL - remove chars from string beginning from specific character

from table I retrieves values, for example,
7752652:1,7752653:2,7752654:3,7752655:4
or
7752941:1,7752942:2
i.e. string may contain any quantity of substrings.
What I need: remove all occurrences of characters from char ':' to a comma char.
For example,
7752652:1,7752653:2,7752654:3,7752655:4
should be
7752652,7752653,7752654,7752655
How do it?
Replace : with start tag <X>.
Replace , with end tag </X> and an extra comma.
Add an extra end tag to the end </X>.
That will give you a string that look like 7752941<X>1</X>,7752942<X>2</X>.
Cast to XML and use query(text()) to get the root text values.
Cast the result back to string.
SQL Fiddle
MS SQL Server 2012 Schema Setup:
create table T
(
C varchar(100)
)
insert into T values
('7752652:1,7752653:2,7752654:3,7752655:4'),
('7752941:1,7752942:2')
Query 1:
select cast(cast(replace(replace(T.C, ':', '<X>'), ',', '</X>,')+'</X>' as xml).query('text()') as varchar(100)) as C
from T
Results:
| C |
|---------------------------------|
| 7752652,7752653,7752654,7752655 |
| 7752941,7752942 |
declare #query varchar(8000)
select #query= 'select '+ replace (
replace('7752652:1,7752653:2,7752654:3,7752655:4',',',' t union all select ')
,':',' t1 , ')
exec(';with cte as ( '+#query+' ) select cast(t1 as varchar)+'','' from cte for xml path('''')')
Try this:
DECLARE #Data VARCHAR(100) = '7752652:1,7752653:2,7752654:3,7752655:4'
DECLARE #Output VARCHAR(100) = ''
WHILE CHARINDEX(':', #Data) > 0
BEGIN
IF LEN(#Output) > 0 SET #Output = #Output + ','
SET #Output = #Output + LEFT(#Data, CHARINDEX(':', #Data)-1)
SET #Data = STUFF(#Data,
1,
(CASE CHARINDEX(',', #Data)
WHEN 0 THEN LEN(#Data)
ELSE CHARINDEX(',', #Data)
END) - CHARINDEX(':', #Data),
'')
END
SELECT #Output AS Result -- 7752652,7752653,7752654,7752655
Hope this will help.
I borrowed the Splitter function from here. You could use any delimiter parser you may already be using.
Parse the string to table values
Used Substring function to remove values after ':'
Use For xml to re-generate CSV
Test Data:'
IF OBJECT_ID(N'tempdb..#temp')>0
DROP TABLE #temp
CREATE TABLE #temp (id int, StringCSV VARCHAR(500))
INSERT INTO #temp VALUES ('1','7752652:1,7752653:2,7752654:3,7752655:4')
INSERT INTO #temp VALUES ('2','7752656:1,7752657:3,7752658:4')
INSERT INTO #temp VALUES ('3','7752659:1,7752660:2')
SELECT * FROM #temp t
Main Query:
;WITH cte_Remove(ID, REMOVE) AS
(
SELECT y.id AS ID,
SUBSTRING(fn.string, 1, CHARINDEX(':', fn.string) -1) AS Removed
FROM #temp AS y
CROSS APPLY dbo.fnParseStringTSQL(y.StringCSV, ',') AS fn
)
SELECT DISTINCT ID,
STUFF(
(
SELECT ',' + REMOVE
FROM cte_Remove AS t2
WHERE t2.ID = t1.ID
FOR XML PATH('')
),1,1,'') AS col2
FROM cte_Remove AS t1
Cleanup Test Data:
IF OBJECT_ID(N'tempdb..#temp') > 0
DROP TABLE #temp
I solved this problem with CLR function. It is more quickly and function can be used in complex queries
public static SqlString fnRemoveSuffics(SqlString source)
{
string pattern = #":(\d+)";
string replacement = "";
string result = Regex.Replace(source.Value, pattern, replacement);
return new SqlString(result);
}

Operand type clash: varchar is incompatible with User-Defined Table Type

I created a User-Defined Table Type:
CREATE TYPE dbo.ListTableType AS TABLE(
ITEM varchar(500) NULL
)
I leverage it in a function:
CREATE FUNCTION dbo.fn_list_to_string
(
#LIST dbo.ListTableType READONLY
)
RETURNS varchar(max)
AS
BEGIN
DECLARE #RESULT varchar(max)
SET #RESULT = ''
DECLARE #NL AS CHAR(2) = CHAR(13) + CHAR(10)
SELECT #RESULT = #RESULT + ITEM + #NL FROM #LIST
SET #RESULT = SUBSTRING(#RESULT, 1, LEN(#RESULT) - 1)
RETURN #RESULT
END
Finally, I try to use this function in a simple select:
SELECT
P.PROGRAM_ID,
PROGRAM_NAME,
PROGRAM_DESC,
P.STATUS_ID,
STATUS_DESC,
P.CONTACT_SID,
I.FIRST_NAME + ' ' + I.LAST_NAME as CONTACT_NAME,
P.CLARITY_ID,
dbo.fn_list_to_string(
( SELECT CONVERT(varchar,CLARITY_ID) as ITEM
FROM dbo.MUSEUM_PROGRAM_PROJECTS as A
JOIN dbo.MUSEUM_PROJECTS as B on B.PROJECT_ID = A.PROJECT_ID
WHERE PROGRAM_ID = P.PROGRAM_ID )
) as PROJECT_CLARITY_IDS
FROM dbo.MUSEUM_PROGRAMS as P
LEFT JOIN dbo.MUSEUM_PROGRAM_STATUS_TYPES as S on S.STATUS_ID = P.STATUS_ID
LEFT JOIN dbo.v_IDVAULT_ENRICHED_CURRENT_EMPLOYEES as I on I.[SID] = P.CONTACT_SID
But I get this error:
Operand type clash: varchar is incompatible with ListTableType
Any idea why? Also if there's another [more elegant] way to achieve what I'm trying to do I'm open to suggestions as well! Thanks in advance!
Here is a simple demonstration of the FOR XML PATH technique which does all of this with a very simple subquery and no table types or extremely inefficient multi-statement table-valued functions etc.
USE tempdb;
GO
CREATE TABLE dbo.P(Program_ID INT);
CREATE TABLE dbo.M(Clarity_ID INT, Program_ID INT);
INSERT dbo.P VALUES(1),(2),(3),(4);
INSERT dbo.M VALUES(1,1),(1,2),(2,3),(3,2),(1,4),(4,1);
SELECT
P.PROGRAM_ID,
PROJECT_CLARITY_IDS = STUFF((
SELECT CHAR(13)+CHAR(10)+CONVERT(VARCHAR(12),Clarity_ID)
FROM dbo.M WHERE Program_ID = p.Program_ID
FOR XML PATH(''), TYPE).value('.[1]','nvarchar(max)'),1,2,'')
FROM dbo.P AS p;
SQLfiddle demo
The output doesn't look right in SQLfiddle or in results to grid in Management Studio, because they strip out carriage returns/line feeds for display purposes, but you can replace CHAR(13)+CHAR(10) with two commas or semi-colons or something to verify that there are two characters there.
Using STUFF..FOR XML PATH construct for concatanation in combination with CTE will get the results you'd like. Something like this:
WITH CTE_PROJECT_CLARITIES AS
(
SELECT DISTINCT PROGRAM_ID
, STUFF((
SELECT CHAR(13) + CHAR(10) + CONVERT(varchar(11),CLARITY_ID)
FROM dbo.MUSEUM_PROGRAM_PROJECTS as A
JOIN dbo.MUSEUM_PROJECTS as B on B.PROJECT_ID = A.PROJECT_ID
WHERE A.PROGRAM_ID = X.PROGRAM_ID
FOR XML PATH ('')),1,2,'') AS PROJECT_CLARITY_IDS
FROM MUSEUM_PROGRAM_PROJECTS X
)
SELECT
P.PROGRAM_ID,
PROGRAM_NAME,
PROGRAM_DESC,
P.STATUS_ID,
STATUS_DESC,
P.CONTACT_SID,
I.FIRST_NAME + ' ' + I.LAST_NAME as CONTACT_NAME,
P.CLARITY_ID,
X.PROJECT_CLARITY_IDS
FROM dbo.MUSEUM_PROGRAMS as P
LEFT JOIN dbo.MUSEUM_PROGRAM_STATUS_TYPES as S on S.STATUS_ID = P.STATUS_ID
LEFT JOIN dbo.v_IDVAULT_ENRICHED_CURRENT_EMPLOYEES as I on I.[SID] = P.CONTACT_SID
LEFT JOIN CTE_PROJECT_CLARITIES X ON X.PROGRAM_ID = p.PROGRAM_ID
SQLFiddle DEMO (not sure if I got the columns right, but you'll get the idea)