I have a table with over 500 columns, dynamically created and named by the user. New columns can be created by the user, but none can be deleted.
I was given the task to program a keyword search that searches through all columns for a specific string, and returns the ID of that record. As you can imagine, the query currently looks something like:
SELECT form_id FROM table_name WHERE col1 LIKE '%str%' OR col2 LIKE '%str%' or col3 LIKE '%str%'.. etc.
It is unbelievably slow. To combat this, I'm trying to create another table, where this data is stored in a different format like this:
form_id, col_name, value
1, 'col2', 'some random value'
1, 'col1', 'another random value'
And then searching using:
SELECT id FROM new_table_name WHERE value LIKE '%str%'
I can export all the data and format it, and insert it into the new table. But how would I go about keeping the new table updated? Is it possible to have triggers that automatically insert/update the new table when the original one is modified? Even though I don't know the column names before hand?
Another option just for fun
Declare #YourTable Table (EmpID int,EmpName varchar(50),Salary int,Location varchar(100))
Insert Into #YourTable Values
(1,'Arul',100,null)
,(2,'Jane',120,'New York')
If 2016+ use JSON
Select*
From #YourTable A
Where (Select A.* For JSON Path,Without_Array_Wrapper ) like '%Jane%'
If <2016 use XML
Select*
From #YourTable A
Where (Select A.* For XML Raw ) like '%Jane%'
Both would Return
EmpID EmpName Salary Location
2 Jane 120 New York
If you want an exact match you can quote the string as such '%"Jane"%'
You can construct the table by unpivoting the original table:
select t.form_id, v.col, v.value
from t cross apply
(values ('col1', col1), ('col2', col2), . . . ) v(col, value);
You can then keep it up-to-date with insert and delete triggers for existing data. Then you will need DDL triggers to handle users adding new columns.
Seems like you are looking for an EAV model.
Here is one approach that does NOT require you to list the 500 columns.
Full Disclosure: This is NOT recommended for HUGE tables. UNPIVOT is more performant.
Also note that if you DON'T want null values remove ,ELEMENTS XSINIL
Example
Declare #YourTable Table (EmpID int,EmpName varchar(50),Salary int,Location varchar(100))
Insert Into #YourTable Values
(1,'Arul',100,null)
,(2,'Jane',120,'New York')
Select Entity = A.EmpID
,C.*
From #YourTable A
Cross Apply ( values (cast((Select A.* for XML RAW,ELEMENTS XSINIL) as xml))) B(XMLData)
Cross Apply (
Select Attribute = a.value('local-name(.)','varchar(100)')
,Value = a.value('.','varchar(max)')
From B.XMLData.nodes('/row') as C1(n)
Cross Apply C1.n.nodes('./*') as C2(a)
) C
Returns
Entity Attribute Value
1 EmpID 1
1 EmpName Arul
1 Salary 100
1 Location <<-- NULL values display as an empty string ... see note regarding nulls
2 EmpID 2
2 EmpName Jane
2 Salary 120
2 Location New York
EDIT - If 2016+ ... JSON
Select A.[EmpID]
,Attribute = B.[Key]
,Value = B.[Value]
From #YourTable A
Cross Apply ( Select * From OpenJson((Select A.* For JSON Path,Without_Array_Wrapper )) ) B
Related
I have a table like this:
TableName
dates
ModelName
BaseUnitPerPallet
pallet
Calendar
June
Null
4
1
Country
June
Null
2
6
Product
June
DOWNSTREAM
Null
8
ProductBOM
June
DOWNSTREAM
9
9
and I want a table like this:
Columns
values
TableName
Calendar
TableName
Country
TableName
Product
TableName
ProductBOM
where columns field is the headers of the previous table, and values are the values in an unpivot way.
I have been trying without success the unpivot logic:
SELECT Columns, Values
FROM
(
SELECT TableName, dates, ModelName, BaseUnitPerPallet, pallet
FROM Database
as source_query
)
UNPIVOT
(
Values FOR Columns IN ( TableName, dates, ModelName, BaseUnitPerPallet, pallete)
)
as pivot_results
any advice or guidance would be great.
Additionally, any resource to do this dinamic? and apply the logic without write the column names?
Thanks in advanceĀ”
I'd recommend using APPLY to unpivot your table
Unpivot using APPLY
DROP TABLE IF EXISTS #YourTable
CREATE TABLE #YourTable (
ID INT IDENTITY(1,1) PRIMARY KEY
,TableName VARCHAR(100)
,Dates Varchar(25)
,ModelName VARCHAR(100)
,BaseUnitPerPallet TINYINT
,Pallet TINYINT
)
INSERT INTO #YourTable
VALUES
('Calendar','June',NULL,4,1)
,('Country','June',NULL,2,6)
,('Product','June','DOWNSTREAM',NULL,8)
,('ProductBOM','June','DOWNSTREAM',9,9)
SELECT A.ID,B.*
FROM #YourTable AS A
CROSS APPLY
(VALUES
('TableName',A.TableName)
,('Dates',A.Dates)
,('ModelName',A.ModelName)
,('BaseUnitPerPallet',CAST(A.BaseUnitPerPallet AS Varchar(100)))
,('Pallet',CAST(A.Pallet AS Varchar(100)))
) AS B(ColumnName,Val)
--WHERE B.Val IS NOT NULL /*Optional in case you want to ignore NULLs*/
ORDER BY A.ID,B.ColumnName
Consider the following query :
DECLARE #T1 TABLE(
[Id] [int] IDENTITY(1,1) NOT NULL,
[Data] VARCHAR(100),
[Column1] VARCHAR(100),
[Column2] VARCHAR(100),
[Column3] VARCHAR(100));
INSERT INTO #T1([Data],[Column1],[Column2],[Column3])
VALUES
('Data1','C11','C21','C31'),
('Data2','C12','C22','C32'),
('Data3','C13','C23','C33'),
('Data4','C14','C24','C34'),
('Data5','C15','C25','C35');
SELECT * FROM #T1;
The output looks like the following:
Now we want to keep the Data column and for each other column stack the result of select for that column into the final table. In other words the following query produces the output:
-- I am looking for a better solution than below!
DECLARE #output TABLE([Data] VARCHAR(100),[Column] VARCHAR(100));
INSERT INTO #output([Data],[Column])
(SELECT [Data],[Column1] FROM #T1
UNION
SELECT [Data],[Column2] FROM #T1
UNION
SELECT [Data],[Column3] FROM #T1)
SELECT * FROM #output
What would be a better cleaner approach than above to produce the final output? As the number of columns increases it means for every single new column I need to have a separate insert which appears to be a crude solution. Ideally I am looking for a pivot-based solution but I couldn't come up with something concrete.
Certainly Yogesh's solution would be more performant. However, since your columns expand over time, here is one approach that will "dynamically" unpivot your data WITHOUT actually using Dynamic SQ:
Example
Select A.[Data]
,C.*
From #T1 A
Cross Apply ( values (cast((Select A.* for XML RAW) as xml))) B(XMLData)
Cross Apply (
Select Item = xAttr.value('local-name(.)', 'varchar(100)')
,Value = xAttr.value('.','varchar(100)')
From XMLData.nodes('//#*') xNode(xAttr)
Where xAttr.value('local-name(.)','varchar(100)') not in ('Id','Data','Other-Columns','To-Exclude')
) C
Returns
I often use apply instead of union :
select t1.data, t2.cols
from #t1 t1 cross apply
( values ([column1]), ([column2]), ([column3]) ) t2(cols);
I have a varchar column, populated by another process where I have no control over, that is filled with comma separated values.
Now I need to find all rows where part of this column exists in that same column, in another row
example
declare #table table (value varchar(50))
insert into #table values ('NB,BD,FR'), ('BD,GK'), ('SL,SR')
select * from #table
so the table contains
value
-----
NB,BD,FR
BD,GK
SL,SR
from the example above I would like to get
value
-----
NB,BD,FR
BD,GK
Because there is a value (in this case BD but can be anything) present in both rows
Can this be done in sql?
You could use clunky XML manipulation to convert comma separated values to rows:
DECLARE #table TABLE (value VARCHAR(50));
INSERT INTO #table VALUES
('NB,BD,FR'),
('BD,GK'),
('SL,SR');
WITH cte AS (
SELECT value, node.value('.', 'varchar(10)') AS substr
FROM #table
CROSS APPLY (SELECT CAST('<x>' + REPLACE(value, ',', '</x>,<x>') + '</x>' AS XML)) AS x(doc)
CROSS APPLY doc.nodes('/x') AS n(node)
)
-- use your favorite technique to find the duplicate
SELECT value
FROM cte AS m
WHERE EXISTS (
SELECT 1
FROM cte AS x
WHERE value <> m.value AND substr = m.substr
)
The CAST(... AS XML) part assumes that your data does not contain characters that have special meaning in XML. The nodes method will convert one row to many, rest is straight forward.
This is the wrong data structure. Don't store values in strings!
declare #table table (id int, value varchar(50));
insert into #table
values (1, 'NB'), (1, 'BD'), (1, 'FR'),
(2, 'BD'), (2, 'GK'),
(3, 'SL'), (3, 'SR');
Then you can get what you want using window functions:
select id, value
from (select t.*, max(cnt) over (partition by id) as max_cnt
from (select t.*, count(*) over (partition by value) as cnt
from #table t
) t
) t
where max_cnt >= 2
I am trying to figure out how to go about getting the values of a comma separated string that's present in one of my cells.
This is the query I current am trying to figure out in my stored procedure:
SELECT
uT.id,
uT.permissions
FROM
usersTbl AS uT
INNER JOIN
usersPermissions AS uP
/*Need to loop here I think?*/
WHERE
uT.active = 'true'
AND
uT.email = 'bbarker#thepriceisright.com'
The usersPermissions table looks like this:
And so a row in the usersTbl table looks like this for permissions:
1,3
I need to find a way to loop through that cell and get each number and place the name ****, in my returned results for the usersTbl.permissions.
So instead of returning this:
Name | id | permissions | age |
------------------------------------
Bbarker | 5987 | 1,3 | 87 |
It needs to returns this:
Name | id | permissions | age |
------------------------------------
Bbarker | 5987 | Read,Upload | 87 |
Really just replacing 1,3 with Read,Upload.
Any help would be great from a SQL GURU!
Reworked query
SELECT
*
FROM
usersTbl AS uT
INNER JOIN
usersPermissionsTbl AS uPT
ON
uPT.userId = uT.id
INNER JOIN
usersPermissions AS uP
ON
uPT.permissionId = uP.id
WHERE
uT.active='true'
AND
uT.email='bBarker#thepriceisright.com'
I agree with all of the comments... but strictly trying to do what you want, here's a way with a splitter function
declare #usersTbl table ([Name] varchar(64), id int, [permissions] varchar(64), age int)
insert into #usersTbl
values
('Bbarker',5987,'1,3',87)
declare #usersTblpermissions table (id int, [type] varchar(64))
insert into #usersTblpermissions
values
(1,'Read'),
(2,'Write'),
(3,'Upload'),
(4,'Admin')
;with cte as(
select
u.[Name]
,u.id as UID
,p.id
,p.type
,u.age
from #usersTbl u
cross apply dbo.DelimitedSplit8K([permissions],',') x
inner join #usersTblpermissions p on p.id = x.Item)
select distinct
[Name]
,UID
,age
,STUFF((
SELECT ',' + t2.type
FROM cte t2
WHERE t.UID = t2.UID
FOR XML PATH(''), TYPE).value('.', 'NVARCHAR(MAX)'), 1, 1, '')
from cte t
Jeff Moden Splitter
CREATE FUNCTION [dbo].[DelimitedSplit8K] (#pString VARCHAR(8000), #pDelimiter CHAR(1))
--WARNING!!! DO NOT USE MAX DATA-TYPES HERE! IT WILL KILL PERFORMANCE!
RETURNS TABLE WITH SCHEMABINDING AS
RETURN
/* "Inline" CTE Driven "Tally Table" produces values from 1 up to 10,000...
enough to cover VARCHAR(8000)*/
WITH E1(N) AS (
SELECT 1 UNION ALL SELECT 1 UNION ALL SELECT 1 UNION ALL
SELECT 1 UNION ALL SELECT 1 UNION ALL SELECT 1 UNION ALL
SELECT 1 UNION ALL SELECT 1 UNION ALL SELECT 1 UNION ALL SELECT 1
), --10E+1 or 10 rows
E2(N) AS (SELECT 1 FROM E1 a, E1 b), --10E+2 or 100 rows
E4(N) AS (SELECT 1 FROM E2 a, E2 b), --10E+4 or 10,000 rows max
cteTally(N) AS (--==== This provides the "base" CTE and limits the number of rows right up front
-- for both a performance gain and prevention of accidental "overruns"
SELECT TOP (ISNULL(DATALENGTH(#pString),0)) ROW_NUMBER() OVER (ORDER BY (SELECT NULL)) FROM E4
),
cteStart(N1) AS (--==== This returns N+1 (starting position of each "element" just once for each delimiter)
SELECT 1 UNION ALL
SELECT t.N+1 FROM cteTally t WHERE SUBSTRING(#pString,t.N,1) = #pDelimiter
),
cteLen(N1,L1) AS(--==== Return start and length (for use in substring)
SELECT s.N1,
ISNULL(NULLIF(CHARINDEX(#pDelimiter,#pString,s.N1),0)-s.N1,8000)
FROM cteStart s
)
--===== Do the actual split. The ISNULL/NULLIF combo handles the length for the final element when no delimiter is found.
SELECT ItemNumber = ROW_NUMBER() OVER(ORDER BY l.N1),
Item = SUBSTRING(#pString, l.N1, l.L1)
FROM cteLen l
;
GO
First, you should read Is storing a delimited list in a database column really that bad?, where you will see a lot of reasons why the answer to this question is Absolutely yes!
Second, you should add a table for user permissions since this is clearly a many to many relationship.
Your tables might look something like this (pseudo code):
usersTbl
(
Id int primary key
-- other user related columns
)
usersPermissionsTbl
(
UserId int, -- Foreign key to usersTbl
PermissionId int, -- Foreign key to permissionsTbl
Primary key (UserId, PermissionId)
)
permissionsTbl
(
Id int primary key,
Name varchar(20)
)
Once you have your tables correct, it's quite easy to get a list of comma separated values from the permissions table.
Adapting scsimon's sample data script to a correct many to many relationship:
declare #users table ([Name] varchar(64), id int, age int)
insert into #users values
('Bbarker',5987,87)
declare #permissions table (id int, [type] varchar(64))
insert into #permissions values
(1,'Read'),
(2,'Write'),
(3,'Upload'),
(4,'Admin')
declare #usersPermissions as table (userId int, permissionId int)
insert into #usersPermissions values (5987, 1), (5987, 3)
Now the query looks like this:
SELECT u.Name,
u.Id,
STUFF(
(
SELECT ','+ [type]
FROM #permissions p
INNER JOIN #usersPermissions up ON p.id = up.permissionId
WHERE up.userId = u.Id
FOR XML PATH('')
)
, 1, 1, '') As Permissions,
u.Age
FROM #Users As u
And the results:
Name Id Permissions Age
Bbarker 5987 Read,Upload 87
You can see a live demo on rextester.
I concur with much of the advice being presented to you in the other responses. The structure you're starting with is not going to be fun to maintain and work with. However, your situation may mean you are stuck with it so maybe some of the tools below will help you.
You can parse the delimiter with charindex() as others demonstrated here- MSSQL - How to split a string using a comma as a separator
... and even better here (several functions are provided) - Split function equivalent in T-SQL?
If you still want to do it with raw inline SQL and are committed to a loop, then pair the string manipulation with a CURSOR. Cursors have their own controversies BTW. The code below will work if your permission syntax remains consistent, which it probably doesn't.
They used charindex(',',columnName) and fed the location into the left() and right() functions along with some additional string evaluation to pull values out. You should be able to piece those together with a cursor
Your query might look like this...
--creating my temp structure
declare #userPermissions table (id int, [type] varchar(16))
insert into #userPermissions (id, [type]) values (1, 'Read')
insert into #userPermissions (id, [type]) values (2, 'Write')
insert into #userPermissions (id, [type]) values (3, 'Upload')
insert into #userPermissions (id, [type]) values (4, 'Admin')
declare #usersTbl table ([Name] varchar(16), id int, [permissions] varchar(8), age int)
insert into #usersTbl ([Name], id, [permissions], age) values ('Bbarker', 5987, '1,3', 87)
insert into #usersTbl ([Name], id, [permissions], age) values ('Mmouse', 5988, '2,4', 88)
--example query
select
ut.[Name]
, (select [type] from #userPermissions where [id] = left(ut.[permissions], charindex(',', ut.[permissions])-1) )
+ ','
+ (select [type] from #userPermissions where [id] = right(ut.[permissions], len(ut.[permissions])-charindex(',', ut.[permissions])) )
from #usersTbl ut
i have a table which contains comma separated values some thing like
id locs
1 a,s,d,f
2 s,d,f,a
3 d,s,a,f
4 d,f,g,a
5 a,s,e
6 f,d
i need out put as 1,2,3,6 in sql server when i have taken comma separated string of id 1.
that means i have taken locs of id 1 and separated with comma, now i want all the ids which contains the separated values of id 1.
Note: I know i don't have to keep comma separated values in table but its happened.
Hope i was clear with my question.
declare #tb table (id int, locs varchar(50))
insert into #tb values(1, 'a,s,d,f'),
(2,'s,d,f,a'),
(3,'d,s,a,f'),
(4,'d,f,g,a'),
(5,'a,s,e'),
(6,'f,d')
declare #cta varchar(20)='s,d,f,a'
;with cte0(id,col2)
as
(
select id,t.c.value('.','varchar(max)') as col2 from (select id,x= cast('<t>'+replace(locs,',','</t><t>') +'</t>' as xml) from #tb) a cross apply x.nodes('/t') t(c)
)
select distinct id from cte0 where #cta like '%'+col2+'%' and id not in( select distinct id from cte0 where #cta not like '%'+col2+'%')
If I understand you correctly, you need to return the id value of all the rows that has at least one of the comma separated values from the locs column of the row you selected. Since this is a poor database design there can only be an ugly solution to this problem.
Start by creating a user defined function to split a comma separated values into a table. there are many ways to do it, this is the first that google found.
DECLARE #Values varchar(max)
SELECT #Values = Locs
FROM Table WHERE Id = #Id
SELECT Id
FROM Table INNER JOIN dbo.Split(#Values) SplitedString
ON( '%,'+ SplitedString.s+',%' LIKE ',' + Locs + ',')