I want to create a stored procedure in SQL Server 2017 and let it be called somewhere else (i.e., Python). It accepts three parameters, stkname - stock name, stktype - stock type, colname - output columns (only these columns are returned). #colname is a varchar storing all required column names separated by ,.
If #colname is not specified, return all columns (*)
If stkname or stktype is not specified, do not filter by it
This is my code so far:
create procedure demo
(#stkname varchar(max),
#colname varchar(max),
#stktype varchar(max))
as
begin
----These lines are pseudo code -----
select
(if #colname is not null, #colname, else *)
from
Datatable
(if #stkname is not null, where stkname = #stkname)
(if #stktype is not null, where stktype = #stktype)
---- pseudo code end here-----
end
The desired result is that
exec demo #colname= 'ticker,price', #stktype = 'A'
returns two columns - ticker and price, for all records with stktype = 'A'
I could imagine 'dynamic SQL' would be possible, but not that 'elegant' and I need to write 2*2*2 = 8 cases.
How can I actually implement it in a better way?
PS: This problem is not a duplicate, since not only I need to pass column names, but also 'generate a query by other parameters'
You need dynamic SQL, but you don't need to write a case for every permutation, and it's really not all that hard to prevent malicious behavior.
CREATE PROCEDURE dbo.demo -- always use schema prefix
#stkname varchar(max) = NULL,
#colnames nvarchar(max) = NULL, -- always use Unicode + proper name
#stktype varchar(max) = NULL
AS
BEGIN
DECLARE #sql nvarchar(max);
SELECT #sql = N'SELECT '
+ STRING_AGG(QUOTENAME(LTRIM(RTRIM(x.value))), ',')
FROM STRING_SPLIT(#colnames, ',') AS x
WHERE EXISTS
(
SELECT 1 FROM sys.columns AS c
WHERE LTRIM(RTRIM(x.value)) = c.name
AND c.[object_id] = OBJECT_ID(N'dbo.DataTable')
);
SET #sql += N' FROM dbo.DataTable WHERE 1 = 1'
+ CASE WHEN #stkname IS NOT NULL THEN
N' AND stkname = #stkname' ELSE N'' END
+ CASE WHEN #stktype IS NOT NULL THEN
N' AND stktype = #stktype' ELSE N'' END;
EXEC sys.sp_executesql #sql,
N'#stkname varchar(max), #stktype varchar(max)',
#stkname, #stktype;
END
It's not clear if the stkname and stktype columns really are varchar(max), I kind of doubt it, you should replace those declarations with the actual data types that match the column definitions. In any case, with this approach the user can't pass nonsense into the column list unless they somehow had the ability to add a column to this table that matched their nonsense pattern (and why are they typing this anyway, instead of picking the columns from a defined list?). And any data they pass as a string to the other two parameters can't possibly be executed here. The thing you are afraid of is probably a result of sloppy code where you carelessly append user input and execute it unchecked, like this:
SET #sql = N'SELECT ' + #weapon_from_user + '...';
For more on malicious behavior see:
Protecting Yourself from SQL Injection in SQL Server - Part 1
Protecting Yourself from SQL Injection in SQL Server - Part 2
I tried this code -
UPDATE Table
SET Name = RTRIM(LTRIM(Name))
Data type of Name is varchar(25)
None of the leading and trailing spaces get removed. When I copy-paste one such Name,
i get this -
"big dash" "space symbol" ABC001
Why is this happening and how do trim the spaces ?
EDIT -
The question has already been answered. I found one more table with this problem. I get
"- value" when i copy a column of a row. When I press the enter key at end of this copy-pasted value, i see more dashes. See image below -
Kindly use below query it will remove space new line etc..
select LTRIM(RTRIM(REPLACE(REPLACE(REPLACE(REPLACE(Name, CHAR(10), CHAR(32)),CHAR(13), CHAR(32)),CHAR(160), CHAR(32)),CHAR(9),CHAR(32))))
I suspect, some non readable(Non-ascii characters) inside the name column, that might not get removed as part of TRIM calls.
select convert(varbinary, Name) from table
Reading the HEX output from above query should reveal the same.
Kindly read this to find how to write functions to remove such characters.
You could do something brute force, such as removing the first character "manually" if it is not alphanumeric:
update table
set name = rtrim(ltrim(case when name not like '[a-zA-Z0-9]%'
then stuff(name, 1, 1, '')
else name
end)
);
You could also search and replace that particular character:
update table
set name = rtrim(ltrim(replace(name, "big dash", '')));
It is a frequent occurrence that we must remove leading and trailing whitespaces from a string before additional processing or sending it to another layer in an application. We can’t always control how the data is entered. The data might come from another system, a data conversion, an old application, EDI, Excel, or from an application which had poor quality control. In some of those cases, a whitespace might not be entered or saved in the system as character 32 which is a whitespace entered in a keyboard. If that happens, SQL built in functions for trimming whitespaces do not work so it becomes necessary to replace the “other” whitespace characters with character 32. Then LTRIM and RTRIM will work as expected.
**Select [udfTrim](ColumnName) from Table**
**CREATE FUNCTION [dbo].[udfTrim]
(
#StringToClean as varchar(8000)
)**
RETURNS varchar(8000)
AS
BEGIN
--Replace all non printing whitespace characers with Characer 32 whitespace
--NULL
Set #StringToClean = Replace(#StringToClean,CHAR(0),CHAR(32));
--Horizontal Tab
Set #StringToClean = Replace(#StringToClean,CHAR(9),CHAR(32));
--Line Feed
Set #StringToClean = Replace(#StringToClean,CHAR(10),CHAR(32));
--Vertical Tab
Set #StringToClean = Replace(#StringToClean,CHAR(11),CHAR(32));
--Form Feed
Set #StringToClean = Replace(#StringToClean,CHAR(12),CHAR(32));
--Carriage Return
Set #StringToClean = Replace(#StringToClean,CHAR(13),CHAR(32));
--Column Break
Set #StringToClean = Replace(#StringToClean,CHAR(14),CHAR(32));
--Non-breaking space
Set #StringToClean = Replace(#StringToClean,CHAR(160),CHAR(32));
Set #StringToClean = LTRIM(RTRIM(#StringToClean));
Return #StringToClean
END
If your string has some non-unicode chars, then those need to be removed first. The functions for that are given later, taken from this link - http://iso30-sql.blogspot.com/2010/10/remove-non-printable-unicode-characters.html
First, check if there are any weird hex chars using -
select convert(varbinary, Name) from table
Then, use the code given in the link above. Note that in the usage of functions, square brackets are to be removed, otherwise the code won't work. Eg. [#DatabaseName = 'MyDatabaseName',] [#SchemaName = 'MySchemaName',]
After this, your strings might have some spaces which can be removed using -
UPDATE Table
SET Name = RTRIM(LTRIM(Name))
Also NOTE that the scripts given in the above link/below will not work on the
following table -
CREATE TABLE [dbo].[Junk](
[JunkHex] nvarchar(50) NULL
) ON [PRIMARY]
GO
GO
INSERT [dbo].[Junk] ([JunkHex]) VALUES (N'Stringğ ')
INSERT [dbo].[Junk] ([JunkHex]) VALUES (N'withħ')
INSERT [dbo].[Junk] ([JunkHex]) VALUES (N'įņvalidđ')
INSERT [dbo].[Junk] ([JunkHex]) VALUES (N'charactersŝ')
This is the content of the link I have given above -
Remove non-printable / Unicode characters in SQL Server 2005
A few months ago, I was upgrading some report templates from the older version of Excel (.xls) to Excel 2007 (.xlsx). I ran into numerous problems almost immediately when I attempted to generate the upgraded reports because the incoming data was riddled with charaters that don't play nicely with XML. The data is used for a variety of reporting purposes, so I decided to tackle the problem on the back-end by removing all but the printable ascii characters.
I started by writing a simple user function for individual strings, but I got to thinking that I may want to automate some of these cleanup tasks and ended up putting something together that allows for a bit more the flexibility. The following creates the basic string user function, along with two procedures to perform the cleanup at the column and table level:
Note - Each of the scripts below uses all the ones above it. So, execute all scripts in order to get all functionality.
Function: fn_npclean_string
use [master]
go
set ansi_nulls on
go
set quoted_identifier on
go
CREATE function [dbo].[fn_npclean_string] (
#strIn as varchar(1000)
)
returns varchar(1000)
as
begin
declare #iPtr as int
set #iPtr = patindex('%[^ -~0-9A-Z]%', #strIn COLLATE LATIN1_GENERAL_BIN)
while #iPtr > 0 begin
set #strIn = replace(#strIn COLLATE LATIN1_GENERAL_BIN, substring(#strIn, #iPtr, 1), '')
set #iPtr = patindex('%[^ -~0-9A-Z]%', #strIn COLLATE LATIN1_GENERAL_BIN)
end
return #strIn
end
Procedure: sp_npclean_col
use [master]
go
set ansi_nulls on
go
set quoted_identifier on
go
CREATE procedure [dbo].[sp_npclean_col]
#DatabaseName varchar(75) = null,
#SchemaName varchar(75) = null,
#TableName varchar(75),
#ColumnName varchar(75)
as
begin
Declare #FullTableName varchar(100)
declare #UpdateSQL nvarchar(1000)
if #DatabaseName is null begin
set #DatabaseName = db_name()
end
if #SchemaName is null begin
set #SchemaName = schema_name()
end
set #FullTableName = '[' + #DatabaseName + '].[' + #SchemaName + '].[' + #TableName + ']'
set #UpdateSQL = 'update ' + #FullTableName + ' set [' + #ColumnName + '] = dbo.fn_npclean_string([' + #ColumnName + ']) where [' + #ColumnName + '] like ''%[^ -~0-9A-Z]%'''
exec sp_ExecuteSQL #UpdateSQL
end
Procedure: sp_npclean_table
use [master]
go
set ansi_nulls on
go
set quoted_identifier on
go
create procedure [dbo].[sp_npclean_table]
#TargetDatabase varchar(75) = null,
#TargetSchema varchar(75) = null,
#TargetTable varchar(75)
as
begin
declare #getColSQL nvarchar(750)
declare #textCol CURSOR
declare #curCol varchar(75)
if #TargetDatabase is null begin
set #TargetDatabase = db_name()
end
if #TargetSchema is null begin
set #TargetSchema = schema_name()
end
set #getColSQL =
'select sc.name
from ' + #TargetDatabase + '.sys.columns sc
join ' + #TargetDatabase + '.sys.types st
on sc.system_type_id = st.system_type_id
join ' + #TargetDatabase + '.sys.objects so
on sc.object_id = so.object_id
join ' + #TargetDatabase + '.sys.schemas ss
on so.schema_id = ss.schema_id
where
so.type = ''U''
and st.name in (''text'',''ntext'',''varchar'',''char'',''nvarchar'',''nchar'')
and sc.is_rowguidcol = 0
and sc.is_identity = 0
and sc.is_computed = 0
and so.name = ''' + #TargetTable + '''
and ss.name = ''' + #TargetSchema + ''''
set #getColSQL = 'set #inCursor = cursor for ' + #getColSQL + ' open #incursor'
execute sp_executesql #getColSQL,N'#inCursor cursor out',#inCursor=#textCol OUT
fetch next from #textCol into #curCol
while ##fetch_status = 0
begin
exec sp_npclean_col #DatabaseName = #TargetDatabase, #SchemaName = #TargetSchema, #TableName = #TargetTable, #ColumnName = #curCol
fetch next from #textCol into #curCol
end
Close #textCol
DeAllocate #textCol
end
Using these, invalid characters can be removed in the following ways:
By String:
select master.dbo.fn_npclean_string('Stringğ withħ įņvalidđ charactersŝ')
By table column:
exec master.dbo.sp_npclean_col [#DatabaseName = 'MyDatabaseName',] [#SchemaName = 'MySchemaName',] #TableName = 'MyTableName', #ColumnName = 'MyColumnName'
By table:
exec master.dbo.sp_npclean_table [#TargetDatabase = 'MyDatabaseName',] [#TargetSchema = 'MySchemaName',] #TargetTable = 'MyTableName'
I use this command while updating all rows' value.
It'll really work out.
For your example:
UPDATE Table SET Name = LTRIM(RTRIM(REPLACE(REPLACE(REPLACE(REPLACE(Name, CHAR(10), CHAR(32)), CHAR(13), CHAR(32)), CHAR(160), CHAR(32)),CHAR(9),CHAR(32))))
You can use the HEX method above, or you can also use the ASCII() function to determine the ASCII code of the character in question...
SELECT ASCII(SUBSTRING(' character string', 1, 1))
SELECT ASCII(SUBSTRING(' character string', 2, 1))
The select only returns 1 value for the character you specify. But it's helpful for determining which ASCII CHAR() value(s) you need to replace.
-Eric Isaacs
Use this to identify the offending character:
select ascii(substring(' Your string with leading invisible character',1,1));
-- returns something like 160
Use this to replace the offending character
replace(' Your string with leading invisible character', char(160),'')
There are cases that the LTRIM RTRIM not doing what you want, to me, it happened because of the tab key when tab key inserted to a database we cant see it in our eyes in this cases trim function doesn't work.
Try this code
UPDATE <TablaName>
SET NAME = CAST(LTRIM(RTRIM(REPLACE(REPLACE(REPLACE(value, CHAR(9), ''), CHAR(13), ''), CHAR(10), ''))) AS VARCHAR(50))
In a stored procedure I dynamically create a temp table by selecting the name of applications from a regular table. Then I add a date column and add the last 12 months. The result looks like this:
So far so good. Now I want to update the data in columns by querying another regular table. Normally it would be something like:
UPDATE ##TempTable
SET [columName] = (SELECT SUM(columName)
FROM RegularTable
WHERE FORMAT(RegularTable.Date,'MM/yyyy') = FORMAT(##TempMonths.x,'MM/yyyy'))
However, since I don't know what the name of the columns are at any given time, I need to do this dynamically.
So my question is, how can I get the column names of a temp table dynamically while doing an update?
Thanks!
I think you can use something like the following.
select name as 'ColumnName'
from tempdb.sys.columns
where object_id = object_id('tempdb..##TempTable');
And then generate dynamic sql using something like the following.
DECLARE #tableName nvarchar(50)
SET #tableName = 'RegularTable'
DECLARE #sql NVARCHAR(MAX)
SET #sql = ''
SELECT #sql = #sql + ' UPDATE ##TempTable ' + CHAR(13) +
' SET [' + c.name + '] = (SELECT SUM([' + c.name + ']) ' + CHAR(13) +
' FROM RegularTable' + CHAR(13) +
' WHERE FORMAT(RegularTable.Date,''MM/yyyy'') = FORMAT(##TempMonths.x,''MM/yyyy''));' + CHAR(13)
from tempdb.sys.columns c
where object_id = object_id('tempdb..##MyTempTable');
print #sql
-- exec sp_executesql #sql;
Then print statement in above snippet shows that the #sql variable has the following text.
UPDATE ##TempTable
SET [Test Application One] = (SELECT SUM([Test Application One])
FROM RegularTable
WHERE FORMAT(RegularTable.Date,'MM/yyyy') = FORMAT(##TempMonths.x,'MM/yyyy'));
UPDATE ##TempTable
SET [Test Application Two] = (SELECT SUM([Test Application Two])
FROM RegularTable
WHERE FORMAT(RegularTable.Date,'MM/yyyy') = FORMAT(##TempMonths.x,'MM/yyyy'));
So now, you use sp_exec to execute the updates as follows (un-comment it from above snippet).
exec sp_executesql #sql;
If it's a 1 time UPDATE you can PRINT the dynamic SQL statement (as shown above) and then execute it in the SSMS Query Windows.
I recommend you use the print statement first to make sure the UPDATE statements generated are what you want, and then do the sp_executesql or run the printed UPDATE statement in the query window.
I'm doing a SQLBulkCopy from my web app and inserting the records into a staging table. This is my first time working with staging tables. The live table that will be accepting the data has about 200 fields and can change into the future. When this change occurs I didn't want to have to re-write the merge statement.
I came up with this SQL that mimics the merge functionality, but doesn't require me to spell out the table columns. I am not an SQL expert and wanted someone that is to take a look and let me know if you see any problems that could arise by using this SQL because I haven't seen any examples of this and many people searching.
Note that records in the staging table that have a null id field are to be inserted.
-- set the table names, primary key field & vars to hold query parts
DECLARE #LiveTable varchar(20) = 'Test'
DECLARE #StagingTable varchar(20) = 'TestStaging'
DECLARE #PKField varchar(20) = 'TestPK'
DECLARE #SQLSet nvarchar(MAX) = ''
DECLARE #SQLInsertFields nvarchar(MAX) = ''
-- get comma delimited field names
DECLARE #Fields nvarchar(MAX) = (SELECT dbo.fn_GetCommaDelimitedFieldNames(#LiveTable))
-- loop through fields generating set clause of query to execute
WHILE LEN(#Fields) > 0
BEGIN
DECLARE #Field varchar(50) = left(#Fields, CHARINDEX(',', #Fields+',')-1)
IF #Field <> #PKField -- the primary key field cannot be updated
BEGIN
SET #SQLSet += ', ' + #LiveTable + '.' + #Field + ' = ' + #StagingTable + '.' + #Field
SET #SQLInsertFields += ', ' + #Field
END
SET #Fields = STUFF(#Fields, 1, CHARINDEX(',', #Fields+','), '')
END
-- remove the leading comma
SET #SQLSet = SUBSTRING(#SQLSet,3,LEN(#SQLSet))
SET #SQLInsertFields = SUBSTRING(#SQLInsertFields,3,LEN(#SQLInsertFields))
-- update records from staging table where primary key is provided
DECLARE #SQL nvarchar(MAX) = N'UPDATE ' + #LiveTable +
' SET ' + #SQLSet +
' FROM ' + #LiveTable +
' INNER JOIN ' + #StagingTable +
' ON ' + #LiveTable + '.' + #PKField + ' = ' + #StagingTable + '.' + #PKField
-- insert records from staging table where primary key is null
SET #SQL += '; INSERT INTO ' + #LiveTable + ' (' + #SQLInsertFields + ') SELECT ' + #SQLInsertFields + ' FROM ' + #StagingTable + ' WHERE ' + #PKField + ' IS NULL'
-- delete the records from the staging table
SET #SQL += '; DELETE FROM ' + #StagingTable
-- execute the sql statement to update existing records and insert new records
exec sp_executesql #SQL;
If anyone see's any issues with performance or anything else, I appreciate the insight.
Don't do this. Really. You're working very hard to avoid a rare problem that you probably won't handle correctly when the time comes.
If the target table changes, how do you know it will change in such a way that your fancy dynamic SQL will work correctly? How can you be sure it won't seem to work -- i.e. will work, syntactically -- but actually do the wrong thing? If and when the target table changes, won't you have to change your application, and the staging table too? With all that in the air, what's adding one more SET clause?
In the meanwhile, how can anyone be expected to read that gobbledygook (not your fault, really, that's SQL's syntax)? A bog-standard insert statement would be very clear and dependable.
And fast. SQL Server can't optimize your dynamic query. You used bcp for efficiency, and now you're defeating it with well meaning futureproofingness.
I have a customer table with Cust_Id, Name, City and search is based upon any or all of the above three.
Which one Should I go for ?
Dynamic SQL:
declare #str varchar(1000)
set #str = 'Select [Sno],[Cust_Id],[Name],[City],[Country],[State]
from Customer where 1 = 1'
if (#Cust_Id != '')
set #str = #str + ' and Cust_Id = ''' + #Cust_Id + ''''
if (#Name != '')
set #str = #str + ' and Name like ''' + #Name + '%'''
if (#City != '')
set #str = #str + ' and City like ''' + #City + '%'''
exec (#str)
Simple query:
select
[Sno],[Cust_Id],[Name],[City],[Country],[State]
from
Customer
where
(#Cust_Id = '' or Cust_Id = #Cust_Id) and
(#Name = '' or Name like #Name + '%') and
(#City = '' or City like #City + '%')
Which one should I prefer (1 or 2) and what are advantages?
After going through everyone's suggestion , here is what i finally got.
DECLARE #str NVARCHAR(1000)
DECLARE #ParametersDefinition NVARCHAR(500)
SET #ParametersDefinition = N'#InnerCust_Id varchar(10),
#InnerName varchar(30),#InnerCity varchar(30)'
SET #str = 'Select [Sno],[Cust_Id],[Name],[City],[Country],[State]
from Customer where 1 = 1'
IF(#Cust_Id != '')
SET #str = #str + ' and Cust_Id = #InnerCust_Id'
IF(#Name != '')
SET #str = #str + ' and Name like #InnerName'
IF(#City != '')
SET #str = #str + ' and City like #InnerCity'
-- ADD the % symbol for search based upon the LIKE keyword
SELECT #Name = #Name + '%', #City = #City+ '%'
EXEC sp_executesql #str, #ParametersDefinition,
#InnerCust_Id = #Cust_Id,
#InnerName = #Name,
#InnerCity = #City;
Note : #Cust_Id, #Name and #City are parameters being passed to the stored procedure
References :
http://blogs.lessthandot.com/index.php/DataMgmt/DataDesign/changing-exec-to-sp_executesql-doesn-t-p
http://www.sommarskog.se/dynamic_sql.html
http://msdn.microsoft.com/en-us/library/ms175170.aspx
Dynamic SQL can be a little more difficult to write, and it is vulnerable to SQL Injection if you are not careful. However, it outperforms the "non-dynamic"/Simple or query.
Read more about it here. http://blogs.lessthandot.com/index.php/DataMgmt/DBProgramming/do-you-use-column-param-or-param-is-null
Dynamic SQL is likley to be more performant which is generally important in a search.
However, it is more diffiult to write and debug and test. First you need to make sure it will not allow SQL injection attacks. Next you need to make sure that the variables you use are large enough to contain the largest possible final SQl statement you would create.
Then you need to create a good number of test cases to make sure that there is not some sort of subtle bug.
You will also need to grant read permissions to the underlying tables which you normally don't need to do if you use Stored procs.
Finally when doing dynamic SQL in a stored proc, please add an input variable called #debug as the last input variable and give it a default value of 0. When a 1 is passed in, instead of executing the dynamic SQL, it will send you the SQL that is created. This will help you debug the proc and and is especially helpful when there is a an error in some future search because you can see exactly what SQL was run for those values.
From my experience, Dynamic SQL makes sense (gains performance) only of decreases the number of JOINs.
Otherwise it only worsen code readability and maintainability.