Related
I created a temp table #test containing 3 fields: ColumnName, TableName, and Id.
I would like to see which rows in the #test table (columns in their respective tables) are not empty? I.e., for every column name that i have in the ColumnName field, and for the corresponding table found in the TableName field, i would like to see whether the column is empty or not. Tried some things (see below) but didn't get anywhere. Help, please.
declare #LoopCounter INT = 1, #maxloopcounter int, #test varchar(100),
#test2 varchar(100), #check int
set #maxloopcounter = (select count(TableName) from #test)
while #LoopCounter <= #maxloopcounter
begin
DECLARE #PropIDs TABLE (tablename varchar(max), id int )
Insert into #PropIDs (tablename, id)
SELECT [tableName], id FROM #test
where id = #LoopCounter
set #test2 = (select columnname from #test where id = #LoopCounter)
declare #sss varchar(max)
set #sss = (select tablename from #PropIDs where id = #LoopCounter)
set #check = (select count(#test2)
from (select tablename
from #PropIDs
where id = #LoopCounter) A
)
print #test2
print #sss
print #check
set #LoopCounter = #LoopCounter + 1
end
In order to use variables as column names and table names in your #Check= query, you will need to use Dynamic SQL.
There is most likely a better way to do this but I cant think of one off hand. Here is what I would do.
Use the select and declare a cursor rather than a while loop as you have it. That way you dont have to count on sequential id's. The cursor would fetch fields columnname, id and tablename
In the loop build a dynamic sql statement
Set #Sql = 'Select Count(*) Cnt Into #Temp2 From ' + TableName + ' Where ' + #columnname + ' Is not null And ' + #columnname <> '''''
Exec(#Sql)
Then check #Temp2 for a value greater than 0 and if this is what you desire you can use the #id that was fetched to update your #Temp table. Putting the result into a scalar variable rather than a temp table would be preferred but cant remember the best way to do that and using a temp table allows you to use an update join so it would well in my opinion.
https://www.mssqltips.com/sqlservertip/1599/sql-server-cursor-example/
http://www.sommarskog.se/dynamic_sql.html
Found a way to extract all non-empty tables from the schema, then just joined with the initial temp table that I had created.
select A.tablename, B.[row_count]
from (select * from #test) A
left join
(SELECT r.table_name, r.row_count, r.[object_id]
FROM sys.tables t
INNER JOIN (
SELECT OBJECT_NAME(s.[object_id]) table_name, SUM(s.row_count) row_count, s.[object_id]
FROM sys.dm_db_partition_stats s
WHERE s.index_id in (0,1)
GROUP BY s.[object_id]
) r on t.[object_id] = r.[object_id]
WHERE r.row_count > 0 ) B
on A.[TableName] = B.[table_name]
WHERE ROW_COUNT > 0
order by b.row_count desc
How about this one - bitmask computed column checks for NULLability. Value in the bitmask tells you if a column is NULL or not. Counting base 2.
CREATE TABLE FindNullComputedMask
(ID int
,val int
,valstr varchar(3)
,NotEmpty as
CASE WHEN ID IS NULL THEN 0 ELSE 1 END
|
CASE WHEN val IS NULL THEN 0 ELSE 2 END
|
CASE WHEN valstr IS NULL THEN 0 ELSE 4 END
)
INSERT FindNullComputedMask
SELECT 1,1,NULL
INSERT FindNullComputedMask
SELECT NULL,2,NULL
INSERT FindNullComputedMask
SELECT 2,NULL, NULL
INSERT FindNullComputedMask
SELECT 3,3,3
SELECT *
FROM FindNullComputedMask
Table1
ID Name Tags
----------------------------------
1 Customer1 Tag1,Tag5,Tag4
2 Customer2 Tag2,Tag6,Tag4,Tag11
3 Customer5 Tag6,Tag5,Tag10
and Table2
ID Name Tags
----------------------------------
1 Product1 Tag1,Tag10,Tag6
2 Product2 Tag2,Tag1,Tag5
3 Product5 Tag1,Tag2,Tag3
what is the best way to join Table1 and Table2 with Tags column?
It should look at the tags column which coma seperated on table 2 for each coma seperated tag on the tags column in the table 1
Note: Tables are not full-text indexed.
The best way is not to have comma separated values in a column. Just use normalized data and you won't have trouble with querying like this - each column is supposed to only have one value.
Without this, there's no way to use any indices, really. Even a full-text index behaves quite different from what you might thing, and they are inherently clunky to use - they're designed for searching for text, not meaningful data. In the end, you will not get much better than something like
where (Col like 'txt,%' or Col like '%,txt' or Col like '%,txt,%')
Using a xml column might be another alternative, though it's still quite a bit silly. It would allow you to treat the values as a collection at least, though.
I don't think there will ever be an easy and efficient solution to this. As Luaan pointed out, it is a very bad idea to store data like this : you lose most of the power of SQL when you squeeze what should be individual units of data into a single cell.
But you can manage this at the slight cost of creating two user-defined functions. First, use this brilliant recursive technique to split the strings into individual rows based on your delimiter :
CREATE FUNCTION dbo.TestSplit (#sep char(1), #s varchar(512))
RETURNS table
AS
RETURN (
WITH Pieces(pn, start, stop) AS (
SELECT 1, 1, CHARINDEX(#sep, #s)
UNION ALL
SELECT pn + 1, stop + 1, CHARINDEX(#sep, #s, stop + 1)
FROM Pieces
WHERE stop > 0
)
SELECT pn AS SplitIndex,
SUBSTRING(#s, start, CASE WHEN stop > 0 THEN stop-start ELSE 512 END) AS SplitPart
FROM Pieces
)
Then, make a function that takes two strings and counts the matches :
CREATE FUNCTION dbo.MatchTags (#a varchar(512), #b varchar(512))
RETURNS INT
AS
BEGIN
RETURN
(SELECT COUNT(*)
FROM dbo.TestSplit(',', #a) a
INNER JOIN dbo.TestSplit(',', #b) b
ON a.SplitPart = b.SplitPart)
END
And that's it, here is a test roll with table variables :
DECLARE #A TABLE (Name VARCHAR(20), Tags VARCHAR(100))
DECLARE #B TABLE (Name VARCHAR(20), Tags VARCHAR(100))
INSERT INTO #A ( Name, Tags )
VALUES
( 'Customer1','Tag1,Tag5,Tag4'),
( 'Customer2','Tag2,Tag6,Tag4,Tag11'),
( 'Customer5','Tag6,Tag5,Tag10')
INSERT INTO #B ( Name, Tags )
VALUES
( 'Product1','Tag1,Tag10,Tag6'),
( 'Product2','Tag2,Tag1,Tag5'),
( 'Product5','Tag1,Tag2,Tag3')
SELECT * FROM #A a
INNER JOIN #B b ON dbo.MatchTags(a.Tags, b.Tags) > 0
I developed a solution as follows:
CREATE TABLE [dbo].[Table1](
Id int not null,
Name nvarchar(250) not null,
Tag nvarchar(250) null,
) ON [PRIMARY]
GO
CREATE TABLE [dbo].[Table2](
Id int not null,
Name nvarchar(250) not null,
Tag nvarchar(250) null,
) ON [PRIMARY]
GO
get sample data for Table1, it will insert 28000 records
INSERT INTO Table1
SELECT CustomerID,CompanyName, (FirstName + ',' + LastName)
FROM AdventureWorks.SalesLT.Customer
GO 3
sample data for Table2.. i need same tags for Table2
declare #tag1 nvarchar(50) = 'Donna,Carreras'
declare #tag2 nvarchar(50) = 'Johnny,Caprio'
get sample data for Table2, it will insert 9735 records
INSERT INTO Table2
SELECT ProductID,Name, (case when(right(ProductID,1)>=5) then #tag1 else #tag2 end)
FROM AdventureWorks.SalesLT.Product
GO 3
My Solution
create TABLE #dt (
Id int IDENTITY(1,1) PRIMARY KEY,
Tag nvarchar(250) NOT NULL
);
I've create temp table and i will fill with Distinct Tag-s in Table1
insert into #dt(Tag)
SELECT distinct Tag
FROM Table1
Now i need to vertical table for tags
create TABLE #Tags ( Tag nvarchar(250) NOT NULL );
Now i'am fill #Tags table with While, you can use Cursor but while is faster
declare #Rows int = 1
declare #Tag nvarchar(1024)
declare #Id int = 0
WHILE #Rows>0
BEGIN
Select Top 1 #Tag=Tag,#Id=Id from #dt where Id>#Id
set #Rows =##RowCount
if #Rows>0
begin
insert into #Tags(Tag) SELECT Data FROM dbo.StringToTable(#Tag, ',')
end
END
last step : join Table2 with #Tags
select distinct t.*
from Table2 t
inner join #Tags on (',' + t.Tag + ',') like ('%,' + #Tags.Tag + ',%')
Table rowcount= 28000 Table2 rowcount=9735 select is less than 2 second
I use this kind of solution with paths of trees. First put a comma at the very begin and at the very end of the string. Than you can call
Where col1 like '%,' || col2 || ',%'
Some database index the column also for the like(postgres do it partially), therefore is also efficient. I don't know sqlserver.
I have a table tbl_Country, which contains columns called ID and Name. The Name column has multiple country names separated by comma, I want the id when I pass multiple country names to compare with Name column values. I am splitting the country names using a function - the sample query looks like this:
#country varchar(50)
SELECT *
FROM tbl_Country
WHERE (SELECT *
FROM Function(#Country)) IN (SELECT *
FROM Function(Name))
tbl_country
ID Name
1 'IN,US,UK,SL,NZ'
2 'IN,PK,SA'
3 'CH,JP'
parameter #country ='IN,SA'
i have to get
ID
1
2
NOTE: The Function will split the string into a datatable
Try this
SELECT * FROM tbl_Country C
LEFT JOIN tbl_Country C1 ON C1.Name=C.Country
Try this:
SELECT *
FROM tbl_Country C
WHERE ',' + #country + ',' LIKE '%,' + C.Name + ',%';
Basically, by specifying multiple values in a single column, you are violating the 1st NF. Therefore, the following might not be a good approach but provides the solution that you are looking for:
declare #country varchar(50)= 'IN,SA'
declare #counterend int
declare #counterstart int =1
declare #singleCountry varchar(10)
set #counterend = (select COUNT(*) from fnSplitStringList(#country))
create table #temp10(
id int
,name varchar(50))
while #counterstart<= #counterend
begin
;with cte as (
select stringliteral country
, ROW_NUMBER() over (order by stringliteral) countryseq
from fnSplitStringList(#country))
select #singleCountry = (select country FROM cte where countryseq=#counterstart)
insert into #temp10(id, name)
select * from tbl_country t1
where not exists (select id from #temp10 t2 where t1.id=t2.id)
and name like '%' + #singleCountry +'%'
set #counterstart= #counterstart+1
end
select * from #temp10
begin drop table #temp10 end
How it works: It splits the passed string and ranks it. Afterwards, it loops through all the records for every single Value(country) produced and inserts them into temptable.
try this,
select a.id FROM tbl_Country a inner join
(SELECT country FROM dbo.Function(#Country)) b on a.name=b.country
My problem is using a table variable in a exec.
declare #sort_col nvarchar(1000) = 'itm_id'
declare #sort_dir nvarchar(4) = 'desc'
declare #filters nvarchar(1000) = ' and itm_name like ''%aa%'''
declare #temp table
(
itm_id int
)
insert into #temp
EXEC('select itm_id from Tblitm where itm_name not like ''%aa%''')
EXEC('select * from (select (ROW_NUMBER() OVER (ORDER BY '+#sort_col+' '+#sort_dir+')) row_num, * FROM (select itm_id, itm_name,
dbo.fnItmsHistory(itm_id) itm_history
from dbo.Tblitm as itm
left outer join '+#temp+' as temp on itm.itm_id = temp.itm_id
where itm_id=itm_id and temp.itm_id = null '+#filters+') as x) as tmp')
It says Must declare the scalar variable "#temp" when the temp table is declared i tried using original temp table and it worked, but i had problems when trying to update my entity model.So is there any solution for this problem?
Note:
I must use exec because in filters i store string for the where clause.
Try moving the table variable inside the dynamic statement.
EXEC('
declare #temp table
(
itm_id int
)
insert into #temp
select itm_id from Tblitm where itm_name not like ''%aa%''
select * from (select (ROW_NUMBER() OVER (ORDER BY '+#sort_col+' '+#sort_dir+')) row_num, * FROM (select itm_id, itm_name,
dbo.fnItmsHistory(itm_id) itm_history
from dbo.Tblitm as itm
left outer join #temp as temp on itm.itm_id = temp.itm_id
where itm_id=itm_id and temp.itm_id = null '+#filters+') as x) as tmp')
For solution i had to use a temp table and then on the start of my stored procedure i used the if condition from the EF can't infer return schema from Stored Procedure selecting from a #temp table anwser.
It's the best solution for this scenario i think.
Consider a column named EmployeeName table Employee. The goal is to delete repeated records, based on the EmployeeName field.
EmployeeName
------------
Anand
Anand
Anil
Dipak
Anil
Dipak
Dipak
Anil
Using one query, I want to delete the records which are repeated.
How can this be done with TSQL in SQL Server?
You can do this with window functions. It will order the dupes by empId, and delete all but the first one.
delete x from (
select *, rn=row_number() over (partition by EmployeeName order by empId)
from Employee
) x
where rn > 1;
Run it as a select to see what would be deleted:
select *
from (
select *, rn=row_number() over (partition by EmployeeName order by empId)
from Employee
) x
where rn > 1;
Assuming that your Employee table also has a unique column (ID in the example below), the following will work:
delete from Employee
where ID not in
(
select min(ID)
from Employee
group by EmployeeName
);
This will leave the version with the lowest ID in the table.
Edit
Re McGyver's comment - as of SQL 2012
MIN can be used with numeric, char, varchar, uniqueidentifier, or datetime columns, but not with bit columns
For 2008 R2 and earlier,
MIN can be used with numeric, char, varchar, or datetime columns, but not with bit columns (and it also doesn't work with GUID's)
For 2008R2 you'll need to cast the GUID to a type supported by MIN, e.g.
delete from GuidEmployees
where CAST(ID AS binary(16)) not in
(
select min(CAST(ID AS binary(16)))
from GuidEmployees
group by EmployeeName
);
SqlFiddle for various types in Sql 2008
SqlFiddle for various types in Sql 2012
You could try something like the following:
delete T1
from MyTable T1, MyTable T2
where T1.dupField = T2.dupField
and T1.uniqueField > T2.uniqueField
(this assumes that you have an integer based unique field)
Personally though I'd say you were better off trying to correct the fact that duplicate entries are being added to the database before it occurs rather than as a post fix-it operation.
DELETE
FROM MyTable
WHERE ID NOT IN (
SELECT MAX(ID)
FROM MyTable
GROUP BY DuplicateColumn1, DuplicateColumn2, DuplicateColumn3)
WITH TempUsers (FirstName, LastName, duplicateRecordCount)
AS
(
SELECT FirstName, LastName,
ROW_NUMBER() OVER (PARTITIONBY FirstName, LastName ORDERBY FirstName) AS duplicateRecordCount
FROM dbo.Users
)
DELETE
FROM TempUsers
WHERE duplicateRecordCount > 1
WITH CTE AS
(
SELECT EmployeeName,
ROW_NUMBER() OVER(PARTITION BY EmployeeName ORDER BY EmployeeName) AS R
FROM employee_table
)
DELETE CTE WHERE R > 1;
The magic of common table expressions.
Try
DELETE
FROM employee
WHERE rowid NOT IN (SELECT MAX(rowid) FROM employee
GROUP BY EmployeeName);
If you're looking for a way to remove duplicates, yet you have a foreign key pointing to the table with duplicates, you could take the following approach using a slow yet effective cursor.
It will relocate the duplicate keys on the foreign key table.
create table #properOlvChangeCodes(
id int not null,
name nvarchar(max) not null
)
DECLARE #name VARCHAR(MAX);
DECLARE #id INT;
DECLARE #newid INT;
DECLARE #oldid INT;
DECLARE OLVTRCCursor CURSOR FOR SELECT id, name FROM Sales_OrderLineVersionChangeReasonCode;
OPEN OLVTRCCursor;
FETCH NEXT FROM OLVTRCCursor INTO #id, #name;
WHILE ##FETCH_STATUS = 0
BEGIN
-- determine if it should be replaced (is already in temptable with name)
if(exists(select * from #properOlvChangeCodes where Name=#name)) begin
-- if it is, finds its id
Select top 1 #newid = id
from Sales_OrderLineVersionChangeReasonCode
where Name = #name
-- replace terminationreasoncodeid in olv for the new terminationreasoncodeid
update Sales_OrderLineVersion set ChangeReasonCodeId = #newid where ChangeReasonCodeId = #id
-- delete the record from the terminationreasoncode
delete from Sales_OrderLineVersionChangeReasonCode where Id = #id
end else begin
-- insert into temp table if new
insert into #properOlvChangeCodes(Id, name)
values(#id, #name)
end
FETCH NEXT FROM OLVTRCCursor INTO #id, #name;
END;
CLOSE OLVTRCCursor;
DEALLOCATE OLVTRCCursor;
drop table #properOlvChangeCodes
delete from person
where ID not in
(
select t.id from
(select min(ID) as id from person
group by email
) as t
);
Please see the below way of deletion too.
Declare #Employee table (EmployeeName varchar(10))
Insert into #Employee values
('Anand'),('Anand'),('Anil'),('Dipak'),
('Anil'),('Dipak'),('Dipak'),('Anil')
Select * from #Employee
Created a sample table named #Employee and loaded it with given data.
Delete aliasName from (
Select *,
ROW_NUMBER() over (Partition by EmployeeName order by EmployeeName) as rowNumber
From #Employee) aliasName
Where rowNumber > 1
Select * from #Employee
Result:
I know, this is asked six years ago, posting just incase it is helpful for anyone.
Here's a nice way of deduplicating records in a table that has an identity column based on a desired primary key that you can define at runtime. Before I start I'll populate a sample data set to work with using the following code:
if exists (select 1 from sys.all_objects where type='u' and name='_original')
drop table _original
declare #startyear int = 2017
declare #endyear int = 2018
declare #iterator int = 1
declare #income money = cast((SELECT round(RAND()*(5000-4990)+4990 , 2)) as money)
declare #salesrepid int = cast(floor(rand()*(9100-9000)+9000) as varchar(4))
create table #original (rowid int identity, monthyear varchar(max), salesrepid int, sale money)
while #iterator<=50000 begin
insert #original
select (Select cast(floor(rand()*(#endyear-#startyear)+#startyear) as varchar(4))+'-'+ cast(floor(rand()*(13-1)+1) as varchar(2)) ), #salesrepid , #income
set #salesrepid = cast(floor(rand()*(9100-9000)+9000) as varchar(4))
set #income = cast((SELECT round(RAND()*(5000-4990)+4990 , 2)) as money)
set #iterator=#iterator+1
end
update #original
set monthyear=replace(monthyear, '-', '-0') where len(monthyear)=6
select * into _original from #original
Next I'll create a Type called ColumnNames:
create type ColumnNames AS table
(Columnnames varchar(max))
Finally I will create a stored proc with the following 3 caveats:
1. The proc will take a required parameter #tablename that defines the name of the table you are deleting from in your database.
2. The proc has an optional parameter #columns that you can use to define the fields that make up the desired primary key that you are deleting against. If this field is left blank, it is assumed that all the fields besides the identity column make up the desired primary key.
3. When duplicate records are deleted, the record with the lowest value in it's identity column will be maintained.
Here is my delete_dupes stored proc:
create proc delete_dupes (#tablename varchar(max), #columns columnnames readonly)
as
begin
declare #table table (iterator int, name varchar(max), is_identity int)
declare #tablepartition table (idx int identity, type varchar(max), value varchar(max))
declare #partitionby varchar(max)
declare #iterator int= 1
if exists (select 1 from #columns) begin
declare #columns1 table (iterator int, columnnames varchar(max))
insert #columns1
select 1, columnnames from #columns
set #partitionby = (select distinct
substring((Select ', '+t1.columnnames
From #columns1 t1
Where T1.iterator = T2.iterator
ORDER BY T1.iterator
For XML PATH ('')),2, 1000) partition
From #columns1 T2 )
end
insert #table
select 1, a.name, is_identity from sys.all_columns a join sys.all_objects b on a.object_id=b.object_id
where b.name = #tablename
declare #identity varchar(max)= (select name from #table where is_identity=1)
while #iterator>=0 begin
insert #tablepartition
Select distinct case when #iterator=1 then 'order by' else 'over (partition by' end ,
substring((Select ', '+t1.name
From #table t1
Where T1.iterator = T2.iterator and is_identity=#iterator
ORDER BY T1.iterator
For XML PATH ('')),2, 5000) partition
From #table T2
set #iterator=#iterator-1
end
declare #originalpartition varchar(max)
if #partitionby is null begin
select #originalpartition = replace(b.value+','+a.type+a.value ,'over (partition by','') from #tablepartition a cross join #tablepartition b where a.idx=2 and b.idx=1
select #partitionby = a.type+a.value+' '+b.type+a.value+','+b.value+') rownum' from #tablepartition a cross join #tablepartition b where a.idx=2 and b.idx=1
end
else
begin
select #originalpartition=b.value +','+ #partitionby from #tablepartition a cross join #tablepartition b where a.idx=2 and b.idx=1
set #partitionby = (select 'OVER (partition by'+ #partitionby + ' ORDER BY'+ #partitionby + ','+b.value +') rownum'
from #tablepartition a cross join #tablepartition b where a.idx=2 and b.idx=1)
end
exec('select row_number() ' + #partitionby +', '+#originalpartition+' into ##temp from '+ #tablename+'')
exec(
'delete a from _original a
left join ##temp b on a.'+#identity+'=b.'+#identity+' and rownum=1
where b.rownum is null')
drop table ##temp
end
Once this is complied, you can delete all your duplicate records by running the proc. To delete dupes without defining a desired primary key use this call:
exec delete_dupes '_original'
To delete dupes based on a defined desired primary key use this call:
declare #table1 as columnnames
insert #table1
values ('salesrepid'),('sale')
exec delete_dupes '_original' , #table1