I have data that is 192 separate columns. They are column pairs so that there is data for a 15 minute time slice and a quality control number. The way it is setup now each row represents a single day. I would like to insert the data into another table with fewer columns (Date,ReadTime,QualityControlNumber,Reading,...)
I started with trying a while loop like this but using a variable to change the column header doesn't seem possible.
Should I nest while loops to increment the column headers or is there another trick I should be using
Code tried:
Declare #count varchar (10),
#QC varchar (10),
#Interval varchar(10)
set #count = 1
set #QC = 'QC#' + #count
set #Interval = 'Interval#' + #count
While (#count<97)
BEGIN
insert into Data_DATEstr (Number,[ReadDate],TimeInterval,QCReading,IntervalReading,ConversionFactor)
select [Number], [Start Date], #count, ['QC#'+#count], [Interval# +#count] ,[Conversion Factor]
from table
where [Number] = '103850581'
and [Start Date] = '060112'
set #count = (#count+1)
END
There is no need to use a while-loop for this. You want an UNPIVOT If you are trying to convert 192 separate columns to rows, then you will definitely want to use an UNPIVOT function. This can be written using dynamic SQL so then you will have have to code all of the fields:
DECLARE #cols AS NVARCHAR(MAX),
#query AS NVARCHAR(MAX)
select #cols = stuff((select ','+quotename(C.name)
from sys.columns as C
where C.object_id = object_id('test') and
C.name like 'col%'
for xml path('')), 1, 1, '')
set #query = 'SELECT QualityControlNumber, replace(col, ''col'', '''') as col, value
from test
unpivot
(
value
for col in (' + #cols + ')
) p '
execute(#query)
See SQL Fiddle with Demo
Using dynamic SQL for this will get the list of fields that you want to transpose when the query is executed. This also prevents you having to manually code for 192 separate columns.
Although your question is not entirely clear, it sounds like you need some form of operation similar to UNPIVOT. This lets you rotate columns into rows. I'd suggest reading up on it and seeing if it will work for you -- even if it doesn't, it might suggest an approach that would work.
The command that you want is unpivot. Something like:
select Date, ReadTime, QualityControlNumber, Reading
from t
unpivot (Reading for date in (day1, . . . , dayn) as unpvt
You will find that you won't really get the date, but instead the original column name. You can fix this by putting the unpivot query in a subquery (or CTE) and then using string manipulations and cast to convert the column name to a date.
Related
I want to build a dynamic SQL query where I can use data from another table as where condition. Let's assume I have two tables: one table with financial data and the other one with conditions. They look something like this:
Table sales
c006 mesocomp c048 c020 c021
----- ---------- ------- ----- ----
120 01TA MICROSOFT 2 239
and a condition table with the following data:
dimension operator wert_db
--------- -------- -------
sales.c006 < 700
sales.c048 not like 'MIC%'
sales.c021 in (203,206)
I want to select all data from sales with the conditions stated in the condition table. So I have an SQL Query as follows:
SELECT *
FROM sales
WHERE sales.c006 < 700
AND sales.c048 NOT LIKE 'MIC%'
AND sales.c021 IN (203, 206)
Since you've posted no attempt to solve or research this yourself, I'll point you in a direction to get you started.
Your question already mentions using Dynamic SQL, so I assume you know at least what that is. You're going to populate a string variable, starting with 'SELECT * FROM Sales '.
You can use the STUFF...FOR XML PATH technique to assemble the conditions rows into a WHERE clause.
One change to the linked example is that you'll need to concatenate dimension, operator and wert_db into one artificial column in the innermost SELECT. Also instead of separating with a comma, you'll separate with ' AND '. And change the parameters of the STUFF function to take off the length of ' AND ' instead of the length of a comma.
DECLARE #tblSales TABLE
(
c006 VARCHAR(10),
mesocomp VARCHAR(100),
c048 VARCHAR(100),
c020 VARCHAR(100),
c021 VARCHAR(100)
)
INSERT INTO #tblSales(c006, mesocomp, c048, c020, c021)
VALUES(120,'01Ta','Microsoft','2','239')
SELECT * FROM #tblSales
DECLARE #tblCondition TABLE
(
Id INT,
dimension VARCHAR(100),
operator VARCHAR(10),
wert_db VARCHAR(100)
)
INSERT INTO #tblCondition(Id, dimension, operator, wert_db) VALUES(1,'sales.c006','<','700')
INSERT INTO #tblCondition(Id, dimension, operator, wert_db) VALUES(1,'sales.c048','not like','''MIC%''')
INSERT INTO #tblCondition(Id, dimension, operator, wert_db) VALUES(1,'sales.c021','in','(203,206)')
DECLARE #whereCondition VARCHAR(400)
SELECT #whereCondition = COALESCE(#whereCondition + ' ', '') + dimension + ' ' + operator + ' ' + wert_db + ' AND '
FROM #tblCondition
SET #whereCondition = SUBSTRING(#whereCondition,0, LEN(#whereCondition) - 3)
PRINT #whereCondition
DECLARE #sql VARCHAR(4000)
SET #sql = 'SELECT * FROM #tblSales Where ' + #whereCondition
PRINT #sql
EXEC(#sql)
--please use real tables so you will get everything working.
Suppose I have a row of data, store such as the following:
------------------------
| Col 1 | Col 2 | Col 3 |
|------------------------|
| Foo | Bar | Foobar |
How might I concatinate this into a single string, such as the below?
Foo-Bar-Foobar
The column headings (and number of column headings) in this table will not be known, so selecting by column name is not an option(?).
Please note that I am not trying to concatinate a list of values in a column, I am trying to concatinate the values stores in one single row. I would also prefer to avoid using pivots, as I will be working with large sets of data and do not want to take the hit to performance.
In such cases I really adore the mighty abilities of XML in dealing with generic sets:
SELECT STUFF(b.query('
for $element in ./*
return
<x>;{$element/text()}</x>
').value('.','nvarchar(max)'),1,1,'')
FROM
(
SELECT TOP 3 * FROM sys.objects o FOR XML PATH('row'),ELEMENTS XSINIL,TYPE
) A(a)
CROSS APPLY a.nodes('/row') B(b);
The result
sysrscols;3;4;0;S ;SYSTEM_TABLE;2017-08-22T19:38:02.860;2017-08-22T19:38:02.867;1;0;0
sysrowsets;5;4;0;S ;SYSTEM_TABLE;2009-04-13T12:59:05.513;2017-08-22T19:38:03.197;1;0;0
sysclones;6;4;0;S ;SYSTEM_TABLE;2017-08-22T19:38:03.113;2017-08-22T19:38:03.120;1;0;0
Remarks
Some things to mention
I use the ; as delimiter, as the - might break with values containing hyphens (e.g. DATE)
I use TOP 3 from sys.objects to create an easy-cheesy-stand-alone sample
Thx to Zohard Peled I added ELEMENTS XSINIL to force the engine not to omit NULL values.
UPDATE Create JSON in pre-2016 versions
You can try this to create a JSON-string in versions before 2016
SELECT '{'
+ STUFF(b.query('
for $element in ./*
return
<x>,"{local-name($element)}":"{$element/text()}"</x>
').value('.','nvarchar(max)'),1,1,'')
+ '}'
FROM
(
SELECT TOP 3 * FROM sys.objects o FOR XML PATH('row'),TYPE
) A(a)
CROSS APPLY a.nodes('/row') B(b);
The result
{"name":"sysrscols","object_id":"3","schema_id":"4","parent_object_id":"0","type":"S ","type_desc":"SYSTEM_TABLE","create_date":"2017-08-22T19:38:02.860","modify_date":"2017-08-22T19:38:02.867","is_ms_shipped":"1","is_published":"0","is_schema_published":"0"}
{"name":"sysrowsets","object_id":"5","schema_id":"4","parent_object_id":"0","type":"S ","type_desc":"SYSTEM_TABLE","create_date":"2009-04-13T12:59:05.513","modify_date":"2017-08-22T19:38:03.197","is_ms_shipped":"1","is_published":"0","is_schema_published":"0"}
{"name":"sysclones","object_id":"6","schema_id":"4","parent_object_id":"0","type":"S ","type_desc":"SYSTEM_TABLE","create_date":"2017-08-22T19:38:03.113","modify_date":"2017-08-22T19:38:03.120","is_ms_shipped":"1","is_published":"0","is_schema_published":"0"}
Hint
You might add ELEMENTS XSINIL to this query as well. This depends, if you'd like NULLs to simply miss, or if you want to include them as "SomeColumn":""
I use UnitE and this is what I would use to select the columns dynamically from the person table.
INFORMATION_SCHEMA.COLUMNS stores the column list for the table and the SELECT statement is built around that.
Declare #Columns NVARCHAR(MAX)
Declare #Table varchar(15) = 'capd_person'
SELECT #Columns=COALESCE(#Columns + ',', '') + COLUMN_NAME
FROM INFORMATION_SCHEMA.COLUMNS
WHERE (TABLE_NAME=#Table )
EXEC('SELECT DISTINCT ' + #Columns + ' FROM ' + #Table)
You would need to change the EXEC command to suit your needs, using CONCAT as described before.
Simply do a SELECT CONCAT(col1,col2,col3) FROM table
However if you wish to make it neat
Use:
SELECT CONCAT(col1,'-',col2,'-',col3) FROM table.
Find more help here.
An improved version of JonTout's answer:
Declare #Columns NVARCHAR(MAX)
Declare #Table varchar(15) = 'TableName'
SELECT #Columns=COALESCE(#Columns + '+', '') +'CONVERT(varchar(max),ISNULL('+ COLUMN_NAME+',''''))+''-'''
FROM INFORMATION_SCHEMA.COLUMNS
WHERE (TABLE_NAME=#Table )
EXEC('SELECT ' + #Columns + ' FROM ' + #Table)
Doing this seemingly trivial task should be simple and obvious using PIVOT - but isn't.
What is the cleanest way to do the conversion, not necessarily using pivot, when limited to ONLY using "pure" SQL (see other factors, below)?
It shouldn't affect the answer, but note that a Python 3.X front end is being used to run SQL queries on a MS SQL Server 2012 backend.
Background :
I need to create CSV files by calling SQL code from Python 3.x. The CSV header line is created from the field (column) names of the SQL table that holds the results of the query.
The following SQL code extracts the field names and returns them as N rows of 1 column - but I need them as 1 row of N columns. (In the example below, the final result must be "A", "B", "C" .)
CREATE TABLE #MyTable -- ideally the real code uses "DECLARE #MyTable TABLE"
(
A varchar( 32 ),
B varchar( 32 ),
C varchar( 32 )
) ;
CREATE TABLE #MetaData -- ideally the real code uses "DECLARE #MetaData TABLE"
(
NameOfField varchar( 32 ) not NULL
) ;
INSERT INTO #MetaData
SELECT name
FROM tempdb.sys.columns as X
WHERE ( object_id = Object_id( 'tempdb..#MyTable' ) )
ORDER BY column_id ; -- generally redundant, ensures correct order if results returned in random order
/*
OK so far, the field names are returned as 3 rows of 1 column (entitled "NameOfField").
Pivoting them into 1 row of 3 columns should be something simple like:
*/
SELECT NameOfField
FROM #MetaData AS Source
PIVOT
(
COUNT( [ NameOfField ] ) FOR [ NameOfField ]
IN ( #MetaData ) -- I've tried "IN (SELECT NameOfField FROM #Metadata)"
) AS Destination ;
This error gets raised twice, once for the COUNT and once for the "FOR" clause of the PIVOT statement:
Msg 207, Level 16, State 1, Line 32
Invalid column name ' NameOfField'.
How do I use the contents of #Metadata to get PIVOT to work? Or is there another simple way?
Other background factors to be aware of:
OBDC (Python's pyodbc package) is being used to pass the SQL queries from - and return the results (a cursor) to - a Python 3.x front end. Consequently there is no opportunity to use any type of manual intervention before the result set is returned to Python.
The above SQL code is intended to become standard boilerplate for every query passed to SQL. The code must dynamically "adapt" itself to the structure of #MyTable (e.g. if field B is removed while D and E are added after C, the end result must be "A", "C","D", "E"). This means that the field names of a table must never appear inside PIVOT's IN clause (the #MetaData table is intended to supply those values).
"Standard" SQL must be used. ALL vendor specific (e.g. Microsoft) extensions/utilities (e.g. "bcp", sqlcmd) must be avoided unless there is a very compelling reason to use them (because "it's there" doesn't count).
For known reasons the select clause (into #Metadata) doesn't work for temporary variables (#MyTable). Is there an equivalent Select that works for temporary variables(i.e. #MetaData)?
UPDATE: This problem is subtly different from that in SQL Server dynamic PIVOT query?. In my case I have to preserve the order of the fields, something not required by that question.
WHY I NEED TO DO THIS:
The python code is a GUI for non-technical people. They use the GUI to pick & chose which (or even all) SQL reports to run from a HUGE number of reports.
Apps like Excel are being used to view these files: to keep our users happy each CSV file must have a header line. The header line will consist of the field names from the SQL table that holds the results of the query.
These scripts can change at any time (e.g. add/delete a column) without any advance notice. To meet our users needs the header line must automatically "adjust itself" to make the corresponding changes. The SQL code below accomplishes this.
The header line gets merged (using UNION) with the query results to form the result set (a cursor) that gets passed back to Python. Python then processes the returned data and creates the CSV file (including the header line) that gets used by our customers.
In a nutshell: We have many sites, many users, many queries. By having SQL "dynmically create" the header line we remove the headache of having to manually manage/coordinate/rollout the SQL changes to all affected parties.
I am unsure what "pure" sql is. Are you refering to ANSI-92 SQL?
Anyhow, if you can use SQL variables, try this:
DECLARE #STRING VARCHAR(MAX)
SELECT #STRING = COALESCE(#STRING + ', ' + '"' + NameOfField + '"', '"' + NameOfField + '"')
FROM #MetaData
SELECT #STRING
/*
Results:
"A", "B", "C"
*/
To #Tab Alleman, thanks. I was able to modify the answer to SQL Server dynamic PIVOT query?
to do the swap (see below) in a way that meets all my needs.
NOTE: For some reason the "DISTINCT" keyword places the fields in alphabetical order - something I don't want.
Commenting that word out (as done below) preserves the order of the fields. I'm a bit uneasy about doing this but in this case it should be safe because the values being selected into #MetaData are guaranteed to be unique.
The difference can be easily seen by swapping fields A & B in #MyTable and uncommenting the "DISTINCT" keyword
--drop table #MyTable
--drop table #MetaData
Create TABLE #MyTable
(
A varchar( 10 ),
B varchar( 10 ),
C varchar( 10 )
)
;
CREATE TABLE #MetaData
(
NameOfField varchar( 100 ) not NULL,
Position int
)
;
INSERT INTO #MetaData
SELECT name, column_id
FROM tempdb.sys.columns as X
WHERE ( object_id = Object_id( 'tempdb..#MyTable' ) )
--ORDER BY column_id -- normally redundant, guards against results being returned in random order
;
select * from #MetaData
DECLARE #cols AS NVARCHAR(MAX),
#query AS NVARCHAR(MAX);
SET #cols = STUFF( (SELECT
-- DISTINCT
',' + QUOTENAME( c.NameOfField )
FROM #MetaData AS c
FOR XML PATH(''), TYPE
).value('.', 'NVARCHAR(MAX)')
,1,1,'')
--print( #cols )
set #query = 'SELECT ' + #cols + ' from
(
select NameOfField
from #MetaData
) AS x
pivot
(
MAX( NameOfField )
for NameOfField in ( '+ #cols + ' )
) AS p
'
--print( #query )
execute( #query )
drop table #MyTable
drop table #MetaData
I have this type of data in a column in my table,,,
QCIM1J77477, 4650125572, QCCR1J77891, 4650312729, QCCR1J74974 --- remove without comma
or
QCIM1E107498,QCIM1E109835,4650092399/ QCCR1E91190, -- remove 4650092399
I want only that string which starts from QC,remove apart from QC,
so please tell me how can I achive it?
Beneath a piece of t-sql script that creates a temporary table #t with temporary variables. Here the temporary table exists untill you break your session, temporary variables exist during the execution of the script. I have a drop table statement at the bottom. Figure out yourself what you want with the table data and whether you want the data put in somewhere else, for example in a not-temporary table :).
I assume you want all the pieces of the string that contain 'QC' as seperate values. If you want your data back as it was originally, that is multiple strings per one column, then you could also do a group by trick. Then you do need a unique identifier of some sort, like name, id, guid of each row or identity.
create table #t ([QCs] nvarchar(100))
declare #str nvarchar(500)
set #str = 'QCIM1E107498,QCIM1E109835,4650092399/ QCCR1E91190'
--replace the above temporary variable with the column you are selecting
declare #sql nvarchar(4000)
select #sql = 'insert into #t select '''+ replace(#str,',',''' union all select ''') + ''''
print #sql
exec ( #sql )
select
QCs
,PATINDEX('%QC%',QCs) as StartPosition
,SUBSTRING(QCs,PATINDEX('%QC%',QCs),12) as QCsNew
from #t where QCs like '%QC%'
drop table #t
With PATINDEX you find the position where in the string 'QC' starts, and with SUBSTRING you tell the dbms to give back (here) 12 characters starting from the found StartPosition.
Beneath what the result looks like. QCsNew is your desired result.
QCs StartPosition QCsNew
QCIM1E107498 1 QCIM1E107498
QCIM1E109835 1 QCIM1E109835
4650092399/ QCCR1E91190 13 QCCR1E91190
I need to select 90 columns out of 107 columns from my table.
Is it possible to write select * except( column1,column2,..) from table or any other way to get specific columns only, or I need to write all the 90 columns in select statement?
You could generate the column list:
select name + ', '
from sys.columns
where object_id = object_id('YourTable')
and name not in ('column1', 'column2')
It's possible to do this on the fly with dynamic SQL:
declare #columns varchar(max)
select #columns = case when #columns is null then '' else #columns + ', ' end +
quotename(name)
from sys.columns
where object_id = object_id('YourTable')
and name not in ('column1', 'column2')
declare #query varchar(max)
set #query = 'select ' + #columns + ' from YourTable'
exec (#query)
No, there's no way of doing * EXCEPT some columns. SELECT * itself should rarely, if ever, be used outside of EXISTS tests.
If you're using SSMS, you can drag the "columns" folder (under a table) from the Object Explorer into a query window, and it will insert all of the column names (so you can then go through them and remove the 17 you don't want)
There is no way in SQL to do select everything EXCEPT col1, col2 etc.
The only way to do this is to have your application handle this, and generate the sql query dynamically.
You could potentially do some dynamic sql for this, but it seems like overkill. Also it's generally considered poor practice to use SELECT *... much less SELECT * but not col3, col4, col5 since you won't get consistent results in the case of table changes.
Just use SSMS to script out a select statement and delete the columns you don't need. It should be simple.
No - you need to write all columns you need. You might create an view for that, so your actual statement could use select * (but then you have to list all columns in the view).
Since you should never be using select *, why is this a problem? Just drag the columns over from the Object Explorer and delete the ones you don't want.