Getting similar column names and count from multiple tables - sql

Question

So I have a table called 'SongsMetadata' in my database with 6 columns as shown below (appx 70k records). It contains all songs related information.
It is slightly different than the regular database table. The 'File_name' column contains .csv files. Those are the actual tables and values in front of them are the columns in that csv file.
So for '1001186_1_7562755270480253254.csv' record in SongsMetadata table, '1001186_1_7562755270480253254' is the table name and it's columns are '&nbsp', 'name', 'album', 'time', 'price' (these tables contain a lot of garbage values)
My goal is to compare all the tables(in this case .csv files) to get all the similar column names and their count. Now I already have a solution to get common column names and count for normal tables here. Each table will be compared with every other table. However, I'm not sure how I can achieve the same with .csv tables.
The expected output is:
1001186_1_7562755270480253254.csv & 1001186_0_5503858345485431752.csv | &nbsp, name, price| 3 #common columns count
1001186_0_5503858345485431752.csv & 99524146_0_3894874701785592836.csv | &nbsp, name, price| 3
and so on...
Any suggestions are appreciated.

Answer 1

The following solution shows how to treat your exsting table so that the wanted matching can occur efficiently, This requires an unpivot although the effect of an unpivot is performed by using cross apply and values which is a simple and efficient method. After that the "matching" is shown, followed by an alternative query for details yo may also find useful. Lastly the new table is displayed just to help visualize what it is.
See the as a live demo at SQL Fiddle
Small Sample:
CREATE TABLE SongsMetadata
([file_name] varchar(7), [col1] varchar(6), [col2] varchar(6), [col3] varchar(6), [col4] varchar(6))
;
INSERT INTO SongsMetadata
([file_name], [col1], [col2], [col3], [col4])
VALUES
('abc.csv', ' ', 'name', 'price', 'artist'),
('def.csv', 'name', ' ', ' ', 'price')
;
UNPIVOT Query
This query moves the column information into a normalized structure to enable the subsequent matching to occur. It is vital to the overall solution. As an added bonus you can mark some column names as "bad" so that these may be ignored later e.g. (which most likely is garbage data)
select
file_name, column_number, column_name
, case when column_name IN (' ','</b>','other-unwanted') then 0 else 1 end as col_is_good
into SongsMetadataUpivot
from (
select file_name, column_number, column_name
from SongsMetadata
cross apply (
values
(1, col1)
, (2, col2)
, (3, col3)
, (4, col4)
) ca (column_number, column_name)
) d
;
Query 1:
This is the "matching logic" provided at http://rextester.com/TLQ28814 but applied to the unpivoted songs data, AND it has the ability to exclude column names you simply don't want to consider (col_is_good).
with fmatch as (
select
l.file_name + ' & ' + r.file_name AS comparing_files
, l.column_name
from SongsMetadataUpivot l
inner join SongsMetadataUpivot r on l.column_name = r.column_name
and l.file_name < r.file_name
and r.col_is_good = 1
where l.col_is_good = 1
)
select --* from fmatch
f.comparing_files
, STUFF((
SELECT
N', ' + column_name
FROM fmatch c
WHERE f.comparing_files = c.comparing_files
order by c.column_name
FOR xml PATH (''), TYPE
)
.value('text()[1]', 'nvarchar(max)'), 1, 2, N'') as columns
, count(*) as num_col_matches
from fmatch f
group by f.comparing_files
Results:
| comparing_files | columns | num_col_matches |
|-------------------|-------------|-----------------|
| abc.csv & def.csv | name, price | 2 |
Query 2:
This will simply allow production of the column lists, in name order, together with their respective column positions in each file.
SELECT
file_name, ca.*
from SongsMetadata f
cross apply (
select
STUFF((
SELECT
N', ' + column_name
FROM SongsMetadataUpivot c
WHERE f.file_name = c.file_name
AND c.col_is_good = 1
ORDER BY column_name
FOR xml PATH (''), TYPE
)
.value('text()[1]', 'nvarchar(max)'), 1, 2, N'')
, STUFF((
SELECT
N', ' + cast(column_number as nvarchar)
FROM SongsMetadataUpivot c
WHERE f.file_name = c.file_name
AND c.col_is_good = 1
ORDER BY column_name
FOR xml PATH (''), TYPE
)
.value('text()[1]', 'nvarchar(max)'), 1, 2, N'')
) ca (column_names, col_numbers)
Results:
| file_name | column_names | col_numbers |
|-----------|---------------------|-------------|
| abc.csv | artist, name, price | 4, 2, 3 |
| def.csv | name, price | 1, 4 |
Query 3:
So you may visualize the "unpivoted" data, the overall solution requires this to occur.
select * from SongsMetadataUpivot
Results:
| file_name | column_number | column_name | col_is_good |
|-----------|---------------|-------------|-------------|
| abc.csv | 1 | | 0 |
| abc.csv | 2 | name | 1 |
| abc.csv | 3 | price | 1 |
| abc.csv | 4 | artist | 1 |
| def.csv | 1 | name | 1 |
| def.csv | 2 | | 0 |
| def.csv | 3 | | 0 |
| def.csv | 4 | price | 1 |