I'd like to pass a table as a parameter into a scaler UDF.
I'd also prefer to restrict the parameter to tables with only one column. (optional)
Is this possible?
EDIT
I don't want to pass a table name, I'd like to pass the table of data (as a reference I presume)
EDIT
I would want my Scaler UDF to basically take a table of values and return a CSV list of the rows.
IE
col1
"My First Value"
"My Second Value"
...
"My nth Value"
would return
"My First Value, My Second Value,... My nth Value"
I'd like to do some filtering on the table though, IE ensuring that there are no nulls and to ensure there are no duplicates. I was expecting something along the lines of:
SELECT dbo.MyFunction(SELECT DISTINCT myDate FROM myTable WHERE myDate IS NOT NULL)
You can, however no any table. From documentation:
For Transact-SQL functions, all data
types, including CLR user-defined
types and user-defined table types,
are allowed except the timestamp data
type.
You can use user-defined table types.
Example of user-defined table type:
CREATE TYPE TableType
AS TABLE (LocationName VARCHAR(50))
GO
DECLARE #myTable TableType
INSERT INTO #myTable(LocationName) VALUES('aaa')
SELECT * FROM #myTable
So what you can do is to define your table type, for example TableType and define the function which takes the parameter of this type. An example function:
CREATE FUNCTION Example( #TableName TableType READONLY)
RETURNS VARCHAR(50)
AS
BEGIN
DECLARE #name VARCHAR(50)
SELECT TOP 1 #name = LocationName FROM #TableName
RETURN #name
END
The parameter has to be READONLY. And example usage:
DECLARE #myTable TableType
INSERT INTO #myTable(LocationName) VALUES('aaa')
SELECT * FROM #myTable
SELECT dbo.Example(#myTable)
Depending on what you want achieve you can modify this code.
EDIT:
If you have a data in a table you may create a variable:
DECLARE #myTable TableType
And take data from your table to the variable
INSERT INTO #myTable(field_name)
SELECT field_name_2 FROM my_other_table
Unfortunately, there is no simple way in SQL Server 2005. Lukasz' answer is correct for SQL Server 2008 though and the feature is long overdue
Any solution would involve temp tables, or passing in xml/CSV and parsing in the UDF. Example: change to xml, parse in udf
DECLARE #psuedotable xml
SELECT
#psuedotable = ...
FROM
...
FOR XML ...
SELECT ... dbo.MyUDF (#psuedotable)
What do you want to do in the bigger picture though? There may be another way to do this...
Edit: Why not pass in the query as a string and use a stored proc with output parameter
Note: this is an untested bit of code, and you'd need to think about SQL injection etc. However, it also satisfies your "one column" requirement and should help you along
CREATE PROC dbo.ToCSV (
#MyQuery varchar(2000),
#CSVOut varchar(max)
)
AS
SET NOCOUNT ON
CREATE TABLE #foo (bar varchar(max))
INSERT #foo
EXEC (#MyQuery)
SELECT
#CSVOut = SUBSTRING(buzz, 2, 2000000000)
FROM
(
SELECT
bar -- maybe CAST(bar AS varchar(max))??
FROM
#foo
FOR XML PATH (',')
) fizz(buzz)
GO
Step 1: Create a Type as Table with name TableType that will accept a table having one varchar column
create type TableType
as table ([value] varchar(100) null)
Step 2: Create a function that will accept above declared TableType as Table-Valued Parameter and String Value as Separator
create function dbo.fn_get_string_with_delimeter (#table TableType readonly,#Separator varchar(5))
returns varchar(500)
As
begin
declare #return varchar(500)
set #return = stuff((select #Separator + value from #table for xml path('')),1,1,'')
return #return
end
Step 3: Pass table with one varchar column to the user-defined type TableType and ',' as separator in the function
select dbo.fn_get_string_with_delimeter(#tab, ',')
Cutting to the bottom line, you want a query like SELECT x FROM y to be passed into a function that returns the values as a comma separated string.
As has already been explained you can do this by creating a table type and passing a UDT into the function, but this needs a multi-line statement.
You can pass XML around without declaring a typed table, but this seems to need a xml variable which is still a multi-line statement i.e.
DECLARE #MyXML XML = (SELECT x FROM y FOR XML RAW);
SELECT Dbo.CreateCSV(#MyXml);
The "FOR XML RAW" makes the SQL give you it's result set as some xml.
But you can bypass the variable using Cast(... AS XML). Then it's just a matter of some XQuery and a little concatenation trick:
CREATE FUNCTION CreateCSV (#MyXML XML)
RETURNS VARCHAR(MAX)
BEGIN
DECLARE #listStr VARCHAR(MAX);
SELECT
#listStr =
COALESCE(#listStr+',' ,'') +
c.value('#Value[1]','nvarchar(max)')
FROM #myxml.nodes('/row') as T(c)
RETURN #listStr
END
GO
-- And you call it like this:
SELECT Dbo.CreateCSV(CAST(( SELECT x FROM y FOR XML RAW) AS XML));
-- Or a working example
SELECT Dbo.CreateCSV(CAST((
SELECT DISTINCT number AS Value
FROM master..spt_values
WHERE type = 'P'
AND number <= 20
FOR XML RAW) AS XML));
As long as you use FOR XML RAW all you need do is alias the column you want as Value, as this is hard coded in the function.
PASSING TABLE AS PARAMETER IN STORED PROCEDURE
Step 1:
CREATE TABLE [DBO].T_EMPLOYEES_DETAILS
(
Id int,
Name nvarchar(50),
Gender nvarchar(10),
Salary int
)
Step 2:
CREATE TYPE EmpInsertType AS TABLE
(
Id int,
Name nvarchar(50),
Gender nvarchar(10),
Salary int
)
Step 3:
/* Must add READONLY keyword at end of the variable */
CREATE PROC PRC_EmpInsertType
#EmployeeInsertType EmpInsertType READONLY
AS
BEGIN
INSERT INTO [DBO].T_EMPLOYEES_DETAILS
SELECT * FROM #EmployeeInsertType
END
Step 4:
DECLARE #EmployeeInsertType EmpInsertType
INSERT INTO #EmployeeInsertType VALUES(1,'John','Male',50000)
INSERT INTO #EmployeeInsertType VALUES(2,'Praveen','Male',60000)
INSERT INTO #EmployeeInsertType VALUES(3,'Chitra','Female',45000)
INSERT INTO #EmployeeInsertType VALUES(4,'Mathy','Female',6600)
INSERT INTO #EmployeeInsertType VALUES(5,'Sam','Male',50000)
EXEC PRC_EmpInsertType #EmployeeInsertType
=======================================
SELECT * FROM T_EMPLOYEES_DETAILS
OUTPUT
1 John Male 50000
2 Praveen Male 60000
3 Chitra Female 45000
4 Mathy Female 6600
5 Sam Male 50000
I've been dealing with a very similar problem and have been able to achieve what I was looking for, even though I'm using SQL Server 2000. I know it is an old question, but think its valid to post here the solution since there should be others like me that use old versions and still need help.
Here's the trick: SQL Server won't accept passing a table to a UDF, nor you can pass a T-SQL query so the function creates a temp table or even calls a stored procedure to do that. So, instead, I've created a reserved table, which I called xtList. This will hold the list of values (1 column, as needed) to work with.
CREATE TABLE [dbo].[xtList](
[List] [varchar](1000) NULL
) ON [PRIMARY]
Then, a stored procedure to populate the list. This is not strictly necessary, but I think is very usefull and best practice.
-- =============================================
-- Author: Zark Khullah
-- Create date: 20/06/2014
-- =============================================
CREATE PROCEDURE [dbo].[xpCreateList]
#ListQuery varchar(2000)
AS
BEGIN
SET NOCOUNT ON;
DELETE FROM xtList
INSERT INTO xtList
EXEC(#ListQuery)
END
Now, just deal with the list in any way you want, using the xtList. You can use in a procedure (for executing several T-SQL commands), scalar functions (for retrieving several strings) or multi-statement table-valued functions (retrieves the strings but like it was inside a table, 1 string per row). For any of that, you'll need cursors:
DECLARE #Item varchar(100)
DECLARE cList CURSOR DYNAMIC
FOR (SELECT * FROM xtList WHERE List is not NULL)
OPEN cList
FETCH FIRST FROM cList INTO #Item
WHILE ##FETCH_STATUS = 0 BEGIN
<< desired action with values >>
FETCH NEXT FROM cList INTO #Item
END
CLOSE cList
DEALLOCATE cList
The desired action would be as follows, depending on which type of object created:
Stored procedures
-- =============================================
-- Author: Zark Khullah
-- Create date: 20/06/2014
-- =============================================
CREATE PROCEDURE [dbo].[xpProcreateExec]
(
#Cmd varchar(8000),
#ReplaceWith varchar(1000)
)
AS
BEGIN
DECLARE #Query varchar(8000)
<< cursor start >>
SET #Query = REPLACE(#Cmd,#ReplaceWith,#Item)
EXEC(#Query)
<< cursor end >>
END
/* EXAMPLES
(List A,B,C)
Query = 'SELECT x FROM table'
with EXEC xpProcreateExec(Query,'x') turns into
SELECT A FROM table
SELECT B FROM table
SELECT C FROM table
Cmd = 'EXEC procedure ''arg''' --whatchout for wrong quotes, since it executes as dynamic SQL
with EXEC xpProcreateExec(Cmd,'arg') turns into
EXEC procedure 'A'
EXEC procedure 'B'
EXEC procedure 'C'
*/
Scalar functions
-- =============================================
-- Author: Zark Khullah
-- Create date: 20/06/2014
-- =============================================
CREATE FUNCTION [dbo].[xfProcreateStr]
(
#OriginalText varchar(8000),
#ReplaceWith varchar(1000)
)
RETURNS varchar(8000)
AS
BEGIN
DECLARE #Result varchar(8000)
SET #Result = ''
<< cursor start >>
SET #Result = #Result + REPLACE(#OriginalText,#ReplaceWith,#Item) + char(13) + char(10)
<< cursor end >>
RETURN #Result
END
/* EXAMPLE
(List A,B,C)
Text = 'Access provided for user x'
with "SELECT dbo.xfProcreateStr(Text,'x')" turns into
'Access provided for user A
Access provided for user B
Access provided for user C'
*/
Multi-statement table-valued functions
-- =============================================
-- Author: Zark Khullah
-- Create date: 20/06/2014
-- =============================================
CREATE FUNCTION [dbo].[xfProcreateInRows]
(
#OriginalText varchar(8000),
#ReplaceWith varchar(1000)
)
RETURNS
#Texts TABLE
(
Text varchar(2000)
)
AS
BEGIN
<< cursor start >>
INSERT INTO #Texts VALUES(REPLACE(#OriginalText,#ReplaceWith,#Item))
<< cursor end >>
END
/* EXAMPLE
(List A,B,C)
Text = 'Access provided for user x'
with "SELECT * FROM dbo.xfProcreateInRow(Text,'x')" returns rows
'Access provided for user A'
'Access provided for user B'
'Access provided for user C'
*/
To obtain the column count on a table, use this:
select count(id) from syscolumns where id = object_id('tablename')
and to pass a table to a function, try XML as show here:
create function dbo.ReadXml (#xmlMatrix xml)
returns table
as
return
( select
t.value('./#Salary', 'integer') as Salary,
t.value('./#Age', 'integer') as Age
from #xmlMatrix.nodes('//row') x(t)
)
go
declare #source table
( Salary integer,
age tinyint
)
insert into #source
select 10000, 25 union all
select 15000, 27 union all
select 12000, 18 union all
select 15000, 36 union all
select 16000, 57 union all
select 17000, 44 union all
select 18000, 32 union all
select 19000, 56 union all
select 25000, 34 union all
select 7500, 29
--select * from #source
declare #functionArgument xml
select #functionArgument =
( select
Salary as [row/#Salary],
Age as [row/#Age]
from #source
for xml path('')
)
--select #functionArgument as [#functionArgument]
select * from readXml(#functionArgument)
/* -------- Sample Output: --------
Salary Age
----------- -----------
10000 25
15000 27
12000 18
15000 36
16000 57
17000 44
18000 32
19000 56
25000 34
7500 29
*/
create table Project (ProjectId int, Description varchar(50));
insert into Project values (1, 'Chase tail, change directions');
insert into Project values (2, 'ping-pong ball in clothes dryer');
create table ProjectResource (ProjectId int, ResourceId int, Name varchar(15));
insert into ProjectResource values (1, 1, 'Adam');
insert into ProjectResource values (1, 2, 'Kerry');
insert into ProjectResource values (1, 3, 'Tom');
insert into ProjectResource values (2, 4, 'David');
insert into ProjectResource values (2, 5, 'Jeff');
SELECT *,
(SELECT Name + ' ' AS [text()]
FROM ProjectResource pr
WHERE pr.ProjectId = p.ProjectId
FOR XML PATH (''))
AS ResourceList
FROM Project p
-- ProjectId Description ResourceList
-- 1 Chase tail, change directions Adam Kerry Tom
-- 2 ping-pong ball in clothes dryer David Jeff
The following will enable you to quickly remove the duplicate,null values and return only the valid one as list.
CREATE TABLE DuplicateTable (Col1 INT)
INSERT INTO DuplicateTable
SELECT 8
UNION ALL
SELECT 1--duplicate
UNION ALL
SELECT 2 --duplicate
UNION ALL
SELECT 1
UNION ALL
SELECT 3
UNION ALL
SELECT 4
UNION ALL
SELECT 5
UNION
SELECT NULL
GO
WITH CTE (COl1,DuplicateCount)
AS
(
SELECT COl1,
ROW_NUMBER() OVER(PARTITION BY COl1 ORDER BY Col1) AS DuplicateCount
FROM DuplicateTable
WHERE (col1 IS NOT NULL)
)
SELECT COl1
FROM CTE
WHERE DuplicateCount =1
GO
CTE are valid in SQL 2005 , you could then store the values in a temp table and use it with your function.
you can do something like this
/* CREATE USER DEFINED TABLE TYPE */
CREATE TYPE StateMaster AS TABLE
(
StateCode VARCHAR(2),
StateDescp VARCHAR(250)
)
GO
/*CREATE FUNCTION WHICH TAKES TABLE AS A PARAMETER */
CREATE FUNCTION TableValuedParameterExample(#TmpTable StateMaster READONLY)
RETURNS VARCHAR(250)
AS
BEGIN
DECLARE #StateDescp VARCHAR(250)
SELECT #StateDescp = StateDescp FROM #TmpTable
RETURN #StateDescp
END
GO
/*CREATE STORED PROCEDURE WHICH TAKES TABLE AS A PARAMETER */
CREATE PROCEDURE TableValuedParameterExample_SP
(
#TmpTable StateMaster READONLY
)
AS
BEGIN
INSERT INTO StateMst
SELECT * FROM #TmpTable
END
GO
BEGIN
/* DECLARE VARIABLE OF TABLE USER DEFINED TYPE */
DECLARE #MyTable StateMaster
/* INSERT DATA INTO TABLE TYPE */
INSERT INTO #MyTable VALUES('11','AndhraPradesh')
INSERT INTO #MyTable VALUES('12','Assam')
/* EXECUTE STORED PROCEDURE */
EXEC TableValuedParameterExample_SP #MyTable
GO
For more details check this link: http://sailajareddy-technical.blogspot.in/2012/09/passing-table-valued-parameter-to.html
Related
I am seeking to create a procedure that I can pass a one column table, and the procedure will output the median. Right now I have a procedure that will determine the median; however, I am getting errors that my #table table variable has not been declared and that the stored procedure could not be found.
My median procedure:
CREATE OR ALTER PROCEDURE dbo.median
(#table NUMERIC,
#median FLOAT OUTPUT)
AS
DECLARE #size AS NUMERIC
SET #size = (SELECT COUNT(*) FROM #table)
SET #median = (SELECT AVG(1) FROM (SELECT *, ROW_NUMBER() OVER(ORDER BY 1) AS ROW FROM #table) AS subquery
WHERE subquery.ROW = ROUND(#size / 2, 0) OR subquery.ROW = ROUND(#size / 2, 0, 1))
RETURN
GO
Calling the procedure:
DECLARE #Arsenic TABLE(Ar FLOAT)
INSERT INTO #Arsenic SELECT Arsenic from dbo.HubspotWaterTestAverages
EXEC dbo.median (SELECT Arsenic FROM dbo.HubspotWaterTestAverages)
NOTE: Arsenic represents the Arsenic level results from water tests, and the values range from null to 10
The working procedure is expected to just return the median value for the column, and later on I am planning on cross joining that to a master table. Thank you for any help!
This may help. Except table variable have to use Table Type.
CREATE TABLE Employee
(
EmpId int NOT NULL,
EmployeeName nvarchar(MAX),
)
GO
CREATE TYPE EmployeeType AS TABLE
(
EmpId int NOT NULL,
EmployeeName nvarchar(MAX)
)
GO
CREATE PROCEDURE PassTableTypeIntoProcedure(#EmployeeType EmployeeType READONLY)
AS
BEGIN
INSERT INTO Employee
SELECT * FROM #EmployeeType
END
GO
DECLARE #EmployeeTypeVariable AS EmployeeType
INSERT INTO #EmployeeTypeVariable VALUES
(1,'A'),
( 2,'B')
EXEC PassTableTypeIntoProcedure #EmployeeTypeVariable
GO
SELECT * FROM Employee
Looking to pass a list of User IDs to return a list names. I have a plan to handle the outputed names (with a COALESCE something or other) but trying to find the best way to pass in the list of user IDs.
The guts of my sproc will look something like this:
create procedure [dbo].[get_user_names]
#user_id_list, --which would equal a list of incoming ID numbers like (5,44,72,81,126)
#username varchar (30) output
as
select last_name+', '+first_name
from user_mstr
where user_id in #user_id_list
Passing the values for #user_id_list is my main concern here.
The preferred method for passing an array of values to a stored procedure in SQL server is to use table valued parameters.
First you define the type like this:
CREATE TYPE UserList AS TABLE ( UserID INT );
Then you use that type in the stored procedure:
create procedure [dbo].[get_user_names]
#user_id_list UserList READONLY,
#username varchar (30) output
as
select last_name+', '+first_name
from user_mstr
where user_id in (SELECT UserID FROM #user_id_list)
So before you call that stored procedure, you fill a table variable:
DECLARE #UL UserList;
INSERT #UL VALUES (5),(44),(72),(81),(126)
And finally call the SP:
EXEC dbo.get_user_names #UL, #username OUTPUT;
As far as I can tell, there are three main contenders: Table-Valued Parameters, delimited list string, and JSON string.
Since 2016, you can use the built-in STRING_SPLIT if you want the delimited route: https://learn.microsoft.com/en-us/sql/t-sql/functions/string-split-transact-sql
That would probably be the easiest/most straightforward/simple approach.
Also since 2016, JSON can be passed as a nvarchar and used with OPENJSON: https://learn.microsoft.com/en-us/sql/t-sql/functions/openjson-transact-sql
That's probably best if you have a more structured data set to pass that may be significantly variable in its schema.
TVPs, it seems, used to be the canonical way to pass more structured parameters, and they are still good if you need that structure, explicitness, and basic value/type checking. They can be a little more cumbersome on the consumer side, though. If you don't have 2016+, this is probably the default/best option.
I think it's a trade off between any of these concrete considerations as well as your preference for being explicit about the structure of your params, meaning even if you have 2016+, you may prefer to explicitly state the type/schema of the parameter rather than pass a string and parse it somehow.
Azure DB, Azure Data WH and from SQL Server 2016, you can use STRING_SPLIT to achieve a similar result to what was described by #sparrow.
Recycling code from #sparrow
WHERE user_id IN (SELECT value FROM STRING_SPLIT( #user_id_list, ',')
Simple and effective way of accepting a list of values into a Stored Procedure
You can try this:
create procedure [dbo].[get_user_names]
#user_id_list varchar(2000), -- You can use any max length
#username varchar (30) output
as
select last_name+', '+first_name
from user_mstr
where user_id in (Select ID from dbo.SplitString( #user_id_list, ',') )
And here is the user defined function for SplitString:
Create FUNCTION [dbo].[SplitString]
(
#Input NVARCHAR(MAX),
#Character CHAR(1)
)
RETURNS #Output TABLE (
Item NVARCHAR(1000)
)
AS
BEGIN
DECLARE #StartIndex INT, #EndIndex INT
SET #StartIndex = 1
IF SUBSTRING(#Input, LEN(#Input) - 1, LEN(#Input)) <> #Character
BEGIN
SET #Input = #Input + #Character
END
WHILE CHARINDEX(#Character, #Input) > 0
BEGIN
SET #EndIndex = CHARINDEX(#Character, #Input)
INSERT INTO #Output(Item)
SELECT SUBSTRING(#Input, #StartIndex, #EndIndex - 1)
SET #Input = SUBSTRING(#Input, #EndIndex + 1, LEN(#Input))
END
RETURN
END
I solved this problem through the following:
In C # I built a String variable.
string userId="";
I put my list's item in this variable. I separated the ','.
for example: in C#
userId= "5,44,72,81,126";
and Send to SQL-Server
SqlParameter param = cmd.Parameters.AddWithValue("#user_id_list",userId);
I Create Separated Function in SQL-server For Convert my Received List (that it's type is NVARCHAR(Max)) to Table.
CREATE FUNCTION dbo.SplitInts
(
#List VARCHAR(MAX),
#Delimiter VARCHAR(255)
)
RETURNS TABLE
AS
RETURN ( SELECT Item = CONVERT(INT, Item) FROM
( SELECT Item = x.i.value('(./text())[1]', 'varchar(max)')
FROM ( SELECT [XML] = CONVERT(XML, '<i>'
+ REPLACE(#List, #Delimiter, '</i><i>') + '</i>').query('.')
) AS a CROSS APPLY [XML].nodes('i') AS x(i) ) AS y
WHERE Item IS NOT NULL
);
In the main Store Procedure, using the command below, I use the entry list.
SELECT user_id = Item FROM dbo.SplitInts(#user_id_list, ',');
this is perfect working for me . this perfect example i hope solved many users problem.
Step 1
Creare reference table in sql like this
Create TYPE dbo.tblNames
AS TABLE
(
[Name] nvarchar(max)
);
go
create TYPE dbo.tblNamesWithCols
AS TABLE
(
[Name] nvarchar(max)
);
go
Step 2
create store procedure with reference table parameters like this
create proc syTest
#VarTbleNameList AS dbo.tblNames READONLY,
#VarTbleNameColsList AS dbo.tblNamesWithCols READONLY,
#VarWhereQuery nvarchar(max)
as
begin
......
...... End
**Calling Store Procedure with parameters **
DECLARE #VarTbleList AS dbo.tblNames
INSERT INTO #VarTbleList
VALUES ( 'tblEmployes' )
INSERT INTO #VarTbleList
VALUES ( 'tblDepartments' )
INSERT INTO #VarTbleList
VALUES ( 'tblCities' )
DECLARE #VarTbleColList AS dbo.tblNamesWithCols
INSERT INTO #VarTbleColList
VALUES ( 'tblEmployes.EmployeId as empId;' )
INSERT INTO #VarTbleColList
VALUES ( 'tblEmployes.EmployeName as empName;' )
INSERT INTO #VarTbleColList
VALUES ( 'tblDepartments.DepartmentName as deptName;' )
INSERT INTO #VarTbleColList
VALUES ( 'tblDepartments.DepartmentId as deptId;' )
EXECUTE syTest #VarTbleList , #VarTbleColList , #VarWhereQuery ='test'
You can use this simple 'inline' method to construct a string_list_type parameter (works in SQL Server 2014):
declare #p1 dbo.string_list_type
insert into #p1 values(N'myFirstString')
insert into #p1 values(N'mySecondString')
Example use when executing a stored proc:
exec MyStoredProc #MyParam=#p1
Check the below code this work for me
#ManifestNoList VARCHAR(MAX)
WHERE
(
ManifestNo IN (SELECT value FROM dbo.SplitString(#ManifestNoList, ','))
)
The proper way is to create a user defined data type:
CREATE TYPE [dbo].[IntArray] AS TABLE
(
[ID] [INT] NULL
)
Then you can use this custom data type:
CREATE OR ALTER PROCEDURE [dbo].[sp_GetUserNames]
(
#userIds [IntArray] READONLY
)
AS
BEGIN
SET NOCOUNT ON;
SELECT
"Name" = u.LastName + ', ' + u.FirstName
FROM dbo.User u
JOIN #userIds uid ON u.Id = uid.Id;
END
Usage:
#DECLARE #result TABLE
(
Name NVARCHAR(max)
);
#DECLARE #ids [IntArray] = SELECT x.userId FROM dbo.sometable x;
SET #result = EXECUTE [dbo].[sp_GetUserNames] #userIds = #ids;
SELECT * FROM #result;
Maybe you could use:
select last_name+', '+first_name
from user_mstr
where ',' + #user_id_list + ',' like '%,' + convert(nvarchar, user_id) + ',%'
I have a web applciation with an iterface that users can uplaod files on. The data form the excel file is collected, concatenated and passed to
a stored procedure which process and returns data.
A brief explanation of the stored procedure.
The stored Procedure collects the string, break it down using a delimeter and stores it in a temp variable table.
Another process is run trough the temp table, where a count is done to find the exact match count and approximate match count by comparing each string
agains a view which contains
all the names to compare against for each row in the first
An exact match count is where the eact string is found in the view for example.. (Bobby Bolonski )
An approximate match is done using a levenshtein distance algorithm database function with a frequency of 2.
temo table #temp1.
The result (name, exactmatch count and approximate match count) are stored in the final temp table.
a select statement is run on the last temp table to return all the data to the application..
MY problem is that, when i passed huge files like and excel file with 27000 names. IT took like 2 hours to process and return data from the database.
I have checked both servers where the application is on and where the database is on.
On the application server. Both memory and cpu usage are less than 15 %
On the database server. both memory and cpu usage are also less than 15 %.
Am looking for advice on what improvements i can do to make the process faster.
Below is the copy of the stored procedure as it is doing all the work and returning the results to the web application.
CREATE PROCEDURE [dbo].[FindMatch]
#fullname varchar(max),#frequency int,
#delimeter varchar(max) AS
set #frequency = 2
declare #transID bigint
SELECT #transID = ABS(CAST(CAST(NEWID() AS VARBINARY(5)) AS Bigint))
DECLARE #exactMatch int = 99
DECLARE #approximateMatch int = 99
declare #name varchar(50)
DECLARE #TEMP1 TABLE (fullname varchar(max),approxMatch varchar(max), exactmatch varchar(max))
DECLARE #ID varchar(max)
--declare a temp table
DECLARE #TEMP TABLE (ID int ,fullname varchar(max),approxMatch varchar(max), exactmatch varchar(max))
--split and store the result in the #temp table
insert into #TEMP (ID,fullname) select * from fnSplitTest(#fullname, #delimeter)
--loop trough the #temp table
WHILE EXISTS (SELECT ID FROM #TEMP)
BEGIN
SELECT Top 1 #ID = ID FROM #TEMP
select #name = fullname from #TEMP where id = #ID
--get the exact match count of the first row from the #temp table and so on until the loop ends
select #exactMatch = count(1) from getalldata where replace(name,',','') COLLATE Latin1_general_CI_AI = #name COLLATE Latin1_general_CI_AI
--declare temp #TEMP3
DECLARE #TEMP3 TABLE (name varchar(max))
--insert into #temp 3 only the data that are similar to our search name so as not to loop over all the data in the view
INSERT INTO #TEMP3(name)
select name from getalldata where SOUNDEX(name) LIKE SOUNDEX(#name)
--get the approximate count using the [DEMLEV] function.
--this function uses the Damerau levenshtein distance algorithm to calculate the distinct between the search string
--and the names inserted into #temp3 above. Uses frequency 2 so as to eliminate all the others
select #approximateMatch = count(1) from #TEMP3 where
dbo.[DamLev](replace(name,',',''),#name,#frequency) <= #frequency and
dbo.[DamLev](replace(name,',',''),#name,#frequency) > 0 and name != #name
--insert into #temp1 at end of every loop results
insert into #TEMP1 (fullname,approxMatch, exactmatch) values(#name,#approximateMatch,#exactMatch)
insert into FileUploadNameInsert (name) values (#name + ' ' +cast(#approximateMatch as varchar) + ' ' + cast(#exactMatch as varchar) + ', ' + cast(#transID as varchar) )
DELETE FROM #TEMP WHERE ID= #ID
delete from #TEMP3
END
--Return all the data stored in #temp3
select fullname,exactmatch,approxMatch, #transID as transactionID from #TEMP1
GO
In my opinion,
Use Openrowset to directly read the records into a pre-defined, properly indexed table of your database.
Now, perform your operations using this table at back-end using pre-defined Stored Procedures.
It should take around 15 minutes for 30,000 rows.
I have a stored procedure that takes in two parameters. I can execute it successfully in Server Management Studio. It shows me the results which are as I expect. However it also returns a Return Value.
It has added this line,
SELECT 'Return Value' = #return_value
I would like the stored procedure to return the table it shows me in the results not the return value as I am calling this stored procedure from MATLAB and all it returns is true or false.
Do I need to specify in my stored procedure what it should return? If so how do I specify a table of 4 columns (varchar(10), float, float, float)?
A procedure can't return a table as such. However you can select from a table in a procedure and direct it into a table (or table variable) like this:
create procedure p_x
as
begin
declare #t table(col1 varchar(10), col2 float, col3 float, col4 float)
insert #t values('a', 1,1,1)
insert #t values('b', 2,2,2)
select * from #t
end
go
declare #t table(col1 varchar(10), col2 float, col3 float, col4 float)
insert #t
exec p_x
select * from #t
I do this frequently using Table Types to ensure more consistency and simplify code. You can't technically return "a table", but you can return a result set and using INSERT INTO .. EXEC ... syntax, you can clearly call a PROC and store the results into a table type. In the following example I'm actually passing a table into a PROC along with another param I need to add logic, then I'm effectively "returning a table" and can then work with that as a table variable.
/****** Check if my table type and/or proc exists and drop them ******/
IF EXISTS (SELECT * FROM sys.objects WHERE type = 'P' AND name = 'returnTableTypeData')
DROP PROCEDURE returnTableTypeData
GO
IF EXISTS (SELECT * FROM sys.types WHERE is_table_type = 1 AND name = 'myTableType')
DROP TYPE myTableType
GO
/****** Create the type that I'll pass into the proc and return from it ******/
CREATE TYPE [dbo].[myTableType] AS TABLE(
[someInt] [int] NULL,
[somenVarChar] [nvarchar](100) NULL
)
GO
CREATE PROC returnTableTypeData
#someInputInt INT,
#myInputTable myTableType READONLY --Must be readonly because
AS
BEGIN
--Return the subset of data consistent with the type
SELECT
*
FROM
#myInputTable
WHERE
someInt < #someInputInt
END
GO
DECLARE #myInputTableOrig myTableType
DECLARE #myUpdatedTable myTableType
INSERT INTO #myInputTableOrig ( someInt,somenVarChar )
VALUES ( 0, N'Value 0' ), ( 1, N'Value 1' ), ( 2, N'Value 2' )
INSERT INTO #myUpdatedTable EXEC returnTableTypeData #someInputInt=1, #myInputTable=#myInputTableOrig
SELECT * FROM #myUpdatedTable
DROP PROCEDURE returnTableTypeData
GO
DROP TYPE myTableType
GO
Consider creating a function which can return a table and be used in a query.
https://msdn.microsoft.com/en-us/library/ms186755.aspx
The main difference between a function and a procedure is that a function makes no changes to any table. It only returns a value.
In this example I'm creating a query to give me the counts of all the columns in a given table which aren't null or empty.
There are probably many ways to clean this up. But it illustrates a function well.
USE Northwind
CREATE FUNCTION usp_listFields(#schema VARCHAR(50), #table VARCHAR(50))
RETURNS #query TABLE (
FieldName VARCHAR(255)
)
BEGIN
INSERT #query
SELECT
'SELECT ''' + #table+'~'+RTRIM(COLUMN_NAME)+'~''+CONVERT(VARCHAR, COUNT(*)) '+
'FROM '+#schema+'.'+#table+' '+
' WHERE isnull("'+RTRIM(COLUMN_NAME)+'",'''')<>'''' UNION'
FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME = #table and TABLE_SCHEMA = #schema
RETURN
END
Then executing the function with
SELECT * FROM usp_listFields('Employees')
produces a number of rows like:
SELECT 'Employees~EmployeeID~'+CONVERT(VARCHAR, COUNT(*)) FROM dbo.Employees WHERE isnull("EmployeeID",'')<>'' UNION
SELECT 'Employees~LastName~'+CONVERT(VARCHAR, COUNT(*)) FROM dbo.Employees WHERE isnull("LastName",'')<>'' UNION
SELECT 'Employees~FirstName~'+CONVERT(VARCHAR, COUNT(*)) FROM dbo.Employees WHERE isnull("FirstName",'')<>'' UNION
You can use an out parameter instead of the return value if you want both a result set and a return value
CREATE PROCEDURE proc_name
#param int out
AS
BEGIN
SET #param = value
SELECT ... FROM [Table] WHERE Condition
END
GO
I had a similar situation and solved by using a temp table inside the procedure, with the same fields being returned by the original Stored Procedure:
CREATE PROCEDURE mynewstoredprocedure
AS
BEGIN
INSERT INTO temptable (field1, field2)
EXEC mystoredprocedure #param1, #param2
select field1, field2 from temptable
-- (mystoredprocedure returns field1, field2)
END
The Status Value being returned by a Stored Procedure can only be an INT datatype. You cannot return other datatypes in the RETURN statement.
From Lesson 2: Designing Stored Procedures:
Every stored procedure can return an integer value known as the
execution status value or return code.
If you still want a table returned from the SP, you'll either have to work the record set returned from a SELECT within the SP or tie into an OUTPUT variable that passes an XML datatype.
HTH,
John
Though this question is very old but as a new in Software Development I can't stop my self to share what I have learnt :D
Creation of Stored Procedure:
CREATE PROC usp_ValidateUSer
(
#UserName nVARCHAR(50),
#Password nVARCHAR(50)
)
AS
BEGIN
IF EXISTS(SELECT '#' FROM Users WHERE Username=#UserName AND Password=#Password)
BEGIN
SELECT u.UserId, u.Username, r.UserRole
FROM Users u
INNER JOIN UserRoles r
ON u.UserRoleId=r.UserRoleId
END
END
Execution of Stored Procedure:
(If you want to test the execution of Stored Procedure in SQL)
EXEC usp_ValidateUSer #UserName='admin', #Password='admin'
The Output:
create procedure PSaleCForms
as
begin
declare
#b varchar(9),
#c nvarchar(500),
#q nvarchar(max)
declare #T table(FY nvarchar(9),Qtr int,title nvarchar (max),invoicenumber nvarchar(max),invoicedate datetime,sp decimal 18,2),grandtotal decimal(18,2))
declare #data cursor
set #data= Cursor
forward_only static
for
select x.DBTitle,y.CurrentFinancialYear from [Accounts Manager].dbo.DBManager x inner join [Accounts Manager].dbo.Accounts y on y.DBID=x.DBID where x.cfy=1
open #data
fetch next from #data
into #c,#b
while ##FETCH_STATUS=0
begin
set #q=N'Select '''+#b+''' [fy], case cast(month(i.invoicedate)/3.1 as int) when 0 then 4 else cast(month(i.invoicedate)/3.1 as int) end [Qtr], l.title,i.invoicenumber,i.invoicedate,i.sp,i.grandtotal from ['+#c+'].dbo.invoicemain i inner join ['+#c+'].dbo.ledgermain l on l.ledgerid=i.ledgerid where (sp=0 or stocktype=''x'') and invoicetype=''DS'''
insert into #T exec [master].dbo.sp_executesql #q
fetch next from #data
into #c,#b
end
close #data
deallocate #data
select * from #T
return
end
Here's an example of a SP that both returns a table and a return value. I don't know if you need the return the "Return Value" and I have no idea about MATLAB and what it requires.
CREATE PROCEDURE test
AS
BEGIN
SELECT * FROM sys.databases
RETURN 27
END
--Use this to test
DECLARE #returnval int
EXEC #returnval = test
SELECT #returnval
Have a Table with the CSV Values in the columns as below
ID Name text
1 SID,DOB 123,12/01/1990
2 City,State,Zip NewYork,NewYork,01234
3 SID,DOB 456,12/21/1990
What is need to get is 2 tables in this scenario as out put with the corresponding values
ID SID DOB
1 123 12/01/1990
3 456 12/21/1990
ID City State Zip
2 NewYork NewYork 01234
Is there any way of achieving it using a Cursor or any other method in SQL server?
There are several ways that this can be done. One way that I would suggest would be to split the data from the comma separated list into multiple rows.
Since you are using SQL Server, you could implement a recursive CTE to split the data, then apply a PIVOT function to create the columns that you want.
;with cte (id, NameItem, Name, textItem, text) as
(
select id,
cast(left(Name, charindex(',',Name+',')-1) as varchar(50)) NameItem,
stuff(Name, 1, charindex(',',Name+','), '') Name,
cast(left(text, charindex(',',text+',')-1) as varchar(50)) textItem,
stuff(text, 1, charindex(',',text+','), '') text
from yt
union all
select id,
cast(left(Name, charindex(',',Name+',')-1) as varchar(50)) NameItem,
stuff(Name, 1, charindex(',',Name+','), '') Name,
cast(left(text, charindex(',',text+',')-1) as varchar(50)) textItem,
stuff(text, 1, charindex(',',text+','), '') text
from cte
where Name > ''
and text > ''
)
select id, SID, DOB
into table1
from
(
select id, nameitem, textitem
from cte
where nameitem in ('SID', 'DOB')
) d
pivot
(
max(textitem)
for nameitem in (SID, DOB)
) piv;
See SQL Fiddle with Demo. The recursive version will work great but if you have a large dataset, you could have some performance issues so you could also use a user defined function to split the data:
create FUNCTION [dbo].[Split](#String1 varchar(MAX), #String2 varchar(MAX), #Delimiter char(1))
returns #temptable TABLE (colName varchar(MAX), colValue varchar(max))
as
begin
declare #idx1 int
declare #slice1 varchar(8000)
declare #idx2 int
declare #slice2 varchar(8000)
select #idx1 = 1
if len(#String1)<1 or #String1 is null return
while #idx1 != 0
begin
set #idx1 = charindex(#Delimiter,#String1)
set #idx2 = charindex(#Delimiter,#String2)
if #idx1 !=0
begin
set #slice1 = left(#String1,#idx1 - 1)
set #slice2 = left(#String2,#idx2 - 1)
end
else
begin
set #slice1 = #String1
set #slice2 = #String2
end
if(len(#slice1)>0)
insert into #temptable(colName, colValue) values(#slice1, #slice2)
set #String1 = right(#String1,len(#String1) - #idx1)
set #String2 = right(#String2,len(#String2) - #idx2)
if len(#String1) = 0 break
end
return
end;
Then you can use a CROSS APPLY to get the result for each row:
select id, SID, DOB
into table1
from
(
select t.id,
c.colname,
c.colvalue
from yt t
cross apply dbo.split(t.name, t.text, ',') c
where c.colname in ('SID', 'DOB')
) src
pivot
(
max(colvalue)
for colname in (SID, DOB)
) piv;
See SQL Fiddle with Demo
You'd need to approach this as a multi-step ETL project. I'd probably start with exporting the two types of rows into a couple staging tables. So, for example:
select * from yourtable /* rows that start with a number */
where substring(text,1,1) in
('0','1','2','3','4','5','6','7','8','9')
select * from yourtable /* rows that don't start with a number */
where substring(text,1,1)
not in ('0','1','2','3','4','5','6','7','8','9')
/* or simply this to follow your example explicitly */
select * from yourtable where name like 'sid%'
select * from yourtable where name like 'city%'
Once you get the two types separated then you can split them out with one of the already written split functions found readily out on the interweb.
Aaron Bertrand (who is on here often) has written up a great post on the variety of ways to split comma delimted strings using SQL. Each of the methods are compared and contrasted here.
http://www.sqlperformance.com/2012/07/t-sql-queries/split-strings
If your row count is minimal (under 50k let's say) and it's going to be a one time operation than pick the easiest way and don't worry too much about all the performance numbers.
If you have a ton of rows or this is an ETL process that will run all the time then you'll really want to pay attention to that stuff.
A simple solution using cursors to build temporary tables. This has the limitation of making all columns VARCHAR and would be slow for large amounts of data.
--** Set up example data
DECLARE #Source TABLE (ID INT, Name VARCHAR(50), [text] VARCHAR(200));
INSERT INTO #Source
(ID, Name, [text])
VALUES (1, 'SID,DOB', '123,12/01/1990')
, (2, 'City,State,Zip', 'NewYork,NewYork,01234')
, (3, 'SID,DOB', '456,12/21/1990');
--** Declare variables
DECLARE #Name VARCHAR(200) = '';
DECLARE #Text VARCHAR(1000) = '';
DECLARE #SQL VARCHAR(MAX);
--** Set up cursor for the tables
DECLARE cursor_table CURSOR FAST_FORWARD READ_ONLY FOR
SELECT s.Name
FROM #Source AS s
GROUP BY Name;
OPEN cursor_table
FETCH NEXT FROM cursor_table INTO #Name;
WHILE ##FETCH_STATUS = 0
BEGIN
--** Dynamically create a temp table with the specified columns
SET #SQL = 'CREATE TABLE ##Table (' + REPLACE(#Name, ',', ' VARCHAR(50),') + ' VARCHAR(50));';
EXEC(#SQL);
--** Set up cursor to insert the rows
DECLARE row_cursor CURSOR FAST_FORWARD READ_ONLY FOR
SELECT s.Text
FROM #Source AS s
WHERE Name = #Name;
OPEN row_cursor;
FETCH NEXT FROM row_cursor INTO #Text;
WHILE ##FETCH_STATUS = 0
BEGIN
--** Dynamically insert the row
SELECT #SQL = 'INSERT INTO ##Table VALUES (''' + REPLACE(#Text, ',', ''',''') + ''');';
EXEC(#SQL);
FETCH NEXT FROM row_cursor INTO #Text;
END
--** Display the table
SELECT *
FROM ##Table;
--** Housekeeping
CLOSE row_cursor;
DEALLOCATE row_cursor;
DROP TABLE ##Table;
FETCH NEXT FROM cursor_table INTO #Name;
END
CLOSE cursor_table;
DEALLOCATE cursor_table;