I need to calculate Min/Max/etc. values for columns in my a table. So I want to filter the columns from INFORMATION_SCHEMA to get only the numeric ones.
The following query returns an error: Error converting data type varchar to int. and I don't know how to fix it.
SELECT #vQuery =
'SELECT '''+TABLE_NAME+''' AS TableName
, '''+COLUMN_NAME+''' AS ColumnName
, '''+DATA_TYPE+''' AS DataType
, MIN(TRY_CAST(TRY_CAST(['+COLUMN_NAME+'] AS VARCHAR(MAX)) AS NUMERIC(30,4))) AS MinValue
, MAX(TRY_CAST(TRY_CAST(['+COLUMN_NAME+'] AS VARCHAR(MAX)) AS NUMERIC(30,4))) AS MaxValue
, AVG(TRY_CAST(TRY_CAST(['+COLUMN_NAME+'] AS VARCHAR(MAX)) AS NUMERIC(30,4))) AS AvgValue
, STDEV(TRY_CAST(TRY_CAST(['+COLUMN_NAME+'] AS VARCHAR(MAX)) AS NUMERIC(30,4))) AS StandardDeviation
, SUM(TRY_CAST(TRY_CAST(['+COLUMN_NAME+'] AS VARCHAR(MAX)) AS NUMERIC(30,4))) AS TotalSum
FROM '+QUOTENAME(TABLE_SCHEMA)+'.'+QUOTENAME(TABLE_NAME)+';'+ CHAR(10)
FROM
(SELECT TABLE_SCHEMA, TABLE_NAME, COLUMN_NAME, DATA_TYPE FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = #Schema
AND TABLE_NAME = #Table
AND DATA_TYPE IN ('BIGINT','NUMERIC','SMALLINT','DECIMAL','SMALLMONEY','INTEGER','INT','TINYINT','MONEY','FLOAT','REAL')) t
This is a bit long for a comment:
Your query appears to work here.
If you were not using the try_() functions, then the problem might occur when you run the query but not when you generate it. And the issue is exponential notation. For instance:
select convert(varchar(255), convert(float, 1234556678990.0))
returns:
'1.23456e+012'
And this value cannot be converted to a numeric value.
I see no advantage to converting to a string before converting to a numeric, so I would drop those conversions. However, with try_cast() this should not be an issue.
In fact, you are generating separate queries for each type. I don't think any conversion is warranted. The queries can return different types if they want. You would need to be concerned about types if you connected the queries using UNION ALL.
Note: You should also use QUOTENAME() on the column names. It can protect against SQL injection, but that seems like a very unlikely problem when you are selecting from INFORMATION_SCHEMA tables (someone would have to create very curious table/column names). However, it also handles non-conforming column names, such as those with spaces.
Related
I'm attempting to validate some column headings before the import of a monthly data set. I've set up an Execute SQL Task that's supposed to retrieve the column headings of the prior month's table and store it in Header_Row as a single string with the field names separated by commas. The query runs just fine in SQL Server, but when running in SSIS, it throws the following error:
"The type of the value (Empty) being assigned to variable 'User:Header_Row' differs from the current variable type (String)."
1) Does this mean that I'm not getting anything back from my query?
2) Is there another method I should be using in SSIS to get the query results I'm looking for?
3) Is there an issue with me using the variable reference in my query as a portion of a string? I think the answer is yes, but would like to confirm, as my variable was still empty after changing this.
Original Query:
SELECT DISTINCT
STUFF((
SELECT
',' + COLUMN_NAME
FROM
db_Analytics.INFORMATION_SCHEMA.COLUMNS aa
WHERE
TABLE_NAME = 'dt_table_?'
ORDER BY
aa.ORDINAL_POSITION
FOR
XML PATH('')
), 1, 1, '') AS Fields
FROM
db_Analytics.INFORMATION_SCHEMA.COLUMNS a;
EDIT: After changing the variable to cover the full table name, I have a new error saying "The value type (__ComObject) can only be converted to variables of the type Object."
Final Query:
SELECT DISTINCT
CAST(STUFF((
SELECT
',' + COLUMN_NAME
FROM
db_Analytics.INFORMATION_SCHEMA.COLUMNS aa
WHERE
TABLE_NAME = ?
ORDER BY
aa.ORDINAL_POSITION
FOR
XML PATH('')
), 1, 1, '') As varchar(8000)) AS Fields
FROM
db_Analytics.INFORMATION_SCHEMA.COLUMNS a;
You are attempting to parameterize your query. Proper query parameterization is useful for avoiding SQL Injection attacks and the like.
Your query is looking for a TABLE_NAME that is literally 'dt_table_?' That's probably not what you want.
For laziness, I'd just rewrite it as
DECLARE #tname sysname = 'dt_table_' + ?;
SELECT DISTINCT
STUFF((
SELECT
',' + COLUMN_NAME
FROM
db_Analytics.INFORMATION_SCHEMA.COLUMNS aa
WHERE
TABLE_NAME = #tname
ORDER BY
aa.ORDINAL_POSITION
FOR
XML PATH('')
), 1, 1, '') AS Fields
FROM
db_Analytics.INFORMATION_SCHEMA.COLUMNS a;
If that's not working, you might need to use an Expression to build out the query.
I'm really pretty sure that this is your problem:
TABLE_NAME = 'dt_table_?'
I'm guessing this is an attempt to parameterize the query, but having the question mark inside the single-quote will cause the question mark to be taken literally.
Try like this instead:
TABLE_NAME = ?
And when you populate the variable that you use as the parameter value, include the 'dt_table_' part in the value of the variable.
EDIT:
Also in your ResultSet assignment, try changing "Fields" to "0" in the Result Name column.
There are two issues with the query above:
1) The query in the task was not properly parameterized. I fixed this by putting the full name of the prior month's table into the variable.
2) The default length of the result was MAX, which was causing an issue when SSIS would try to put it into my variable, Header_Row. I fixed this by casting the result of the query as varchar(8000).
Thanks for the help everyone.
One column in one table of my database say (Table A and Column A) can be either of Numeric type or VARCHAR type. Datatype is decided dynamically and then table gets created.
I need to create a dynamic variable (#Dynamic) which should check the datatype of this column and assign a different column (column B or column C) to it accordingly i.e.
If column A is NVARCHAR, assign column B to #Dynamic
If column A is NUMERIC, assign column C to #Dynamic
I've to do this in both SQL Server and Oracle.
Any help to write a function for this would be greatly appreciated.
In sql server you can check column data type
SELECT DATA_TYPE
FROM INFORMATION_SCHEMA.COLUMNS
WHERE
TABLE_NAME = 'Table A'
AND COLUMN_NAME = 'Column A'
In oracle
SELECT Type
FROM user_tab_columns
WHERE table_name = 'Table A'
AND column_name = 'Column A';
Sql_variant is a dynamic type in SQL Server. The eqivalent in oracle would be anydata type.
But if you are using dynamic SQL why don't you store the data in nvarchar as it is big enough for the convertet numeric values as well and you can use it directly for your dynamic SQL statement?
Actually this was a generic question and did not need any sample data to answer.
The approach I am following is:
I wrote a function
CREATE FUNCTION [dbo].[GET_DATA_TYPE] (#input VARCHAR)
RETURNS VARCHAR(255) AS
BEGIN
DECLARE #l_data_type VARCHAR(255)
SELECT #l_data_type=DATA_TYPE
FROM INFORMATION_SCHEMA.COLUMNS
WHERE
TABLE_NAME = #input AND
COLUMN_NAME = 'AssetID'
RETURN #l_data_type
END
I would call this function in my stored procedures as
SELECT #dataType = dbo.GET_DATA_TYPE(#input);
I declared another variable i.e. #FinalType
IF #dataType == 'numeric'
THEN #FinalType = columnC
IF #dataType == 'nvarchar'
THEN #FinalType = columnD
Then I'll use this #FinalType variable in all my dynamic sqls.
Any other efficient way to do this.
Assuming that we are trying to alter the type of a column in a SQL table, say from varchar to float, using: ALTER TABLE <mytable. ALTER COLUMN <mycolumn> FLOAT. However, we get the error Error to convert datatype varchar to float.
Is it possible to narrow down the cells in the column that are causing this problem?
Thanks,
You can use the ISNUMERIC function:
select * from table where isnumeric(mycolumn) = 0
If you allow NULL values in your column, you'll also need to add a check for NULLs since ISNUMERIC(NULL) evaluates to 0 as well
select * from table where isnumeric(mycolumn) = 0 or mycolumn is not null
I have encounter the same issue while writing ETL procedure. moving staging data into actual core table and we had all columns on staging table a NVARCHAR.
there could be a numeric value which is either scientific format (like very large float values in Excel cell) or it has one of this special CHAR in it. ISNUMERIC function evaluates this char as True when it is appear as whole value.
for example
SELECT ISUMERIC('$'), ISNUMERIC('.')
so just check if any of cell in that column has such values.
'$'
'-'
'+'
','
'.'
if you find that cell has one of above then just exclude such data in your query.
if you find that you have data in scientific format like "1.2408E+12" then ISNUMERIC will be still evaluate it as TRUE but straight insert will fail so convert in appropriate numeric format.
DECLARE #t NUMERIC(28,10)
SELECT #t=CONVERT(NUMERIC(28,10),CONVERT(FLOAT,'1.2408E+12'))
SELECT #t
Dirty, but effective. This removes all characters found in floats (#s and decimal - I'm US-centric). The result you get from the query are items that would need to be reviewed to determine what should be done (ie the cells causing you problems).
SELECT
*
FROM (
SELECT
TableId
, REPLACE(
REPLACE(
REPLACE(
REPLACE(
REPLACE(
REPLACE(
REPLACE(
REPLACE(
REPLACE(
REPLACE(
REPLACE(
ISNULL(Col1,'')
,'0','')
,'1','')
,'2','')
,'3','')
,'4','')
,'5','')
,'6','')
,'7','')
,'8','')
,'9','')
,'.','') [FilteredCol1]
FROM Table
) a
WHERE len(a.[FilteredCol1])>0
Select any records where the varchar value contains any non-numeric characters
SELECT col
FROM tab
WHERE col LIKE '%[^0-9.]%'
and any rows that might have more than one period:
SELECT col
FROM tab
WHERE col LIKE '%.%.%'
I want to describe a table in SQL Server 2008 like what we can do with the DESC command in Oracle.
I have table [EX].[dbo].[EMP_MAST] which I want to describe, but it does not work.
Error shown:
The object 'EMP_MAST' does not exist in database 'master' or is
invalid for this operation.
You can use sp_columns, a system stored procedure for describing a table.
exec sp_columns TableName
You can also use sp_help.
According to this documentation:
DESC MY_TABLE
is equivalent to
SELECT column_name "Name", nullable "Null?",
concat(concat(concat(data_type,'('),data_length),')') "Type" FROM
user_tab_columns WHERE table_name='TABLE_NAME_TO_DESCRIBE';
I've roughly translated that to the SQL Server equivalent for you - just make sure you're running it on the EX database.
SELECT column_name AS [name],
IS_NULLABLE AS [null?],
DATA_TYPE + COALESCE('(' + CASE WHEN CHARACTER_MAXIMUM_LENGTH = -1
THEN 'Max'
ELSE CAST(CHARACTER_MAXIMUM_LENGTH AS VARCHAR(5))
END + ')', '') AS [type]
FROM INFORMATION_SCHEMA.Columns
WHERE table_name = 'EMP_MAST'
The sp_help built-in procedure is the SQL Server's closest thing to Oracle's DESC function IMHO
sp_help MyTable
Use
sp_help "[SchemaName].[TableName]"
or
sp_help "[InstanceName].[SchemaName].[TableName]"
in case you need to qualify the table name further
You can use keyboard short-cut for Description/ detailed information of Table in SQL Server 2008.
Follow steps:
Write Table Name,
Select it, and press Alt + F1
It will show detailed information/ description of mentioned table as,
1) Table created date,
2) Columns Description,
3) Identity,
4) Indexes,
5) Constraints,
6) References etc. As shown Below [example]:
May be this can help:
Use MyTest
Go
select * from information_schema.COLUMNS where TABLE_NAME='employee'
{ where: MyTest= DatabaseName
Employee= TableName } --Optional conditions
I like the answer that attempts to do the translate, however, while using the code it doesn't like columns that are not VARCHAR type such as BIGINT or DATETIME. I needed something similar today so I took the time to modify it more to my liking. It is also now encapsulated in a function which is the closest thing I could find to just typing describe as oracle handles it. I may still be missing a few data types in my case statement but this works for everything I tried it on. It also orders by ordinal position. this could be expanded on to include primary key columns easily as well.
CREATE FUNCTION dbo.describe (#TABLENAME varchar(50))
returns table
as
RETURN
(
SELECT TOP 1000 column_name AS [ColumnName],
IS_NULLABLE AS [IsNullable],
DATA_TYPE + '(' + CASE
WHEN DATA_TYPE = 'varchar' or DATA_TYPE = 'char' THEN
CASE
WHEN Cast(CHARACTER_MAXIMUM_LENGTH AS VARCHAR(5)) = -1 THEN 'Max'
ELSE Cast(CHARACTER_MAXIMUM_LENGTH AS VARCHAR(5))
END
WHEN DATA_TYPE = 'decimal' or DATA_TYPE = 'numeric' THEN
Cast(NUMERIC_PRECISION AS VARCHAR(5))+', '+Cast(NUMERIC_SCALE AS VARCHAR(5))
WHEN DATA_TYPE = 'bigint' or DATA_TYPE = 'int' THEN
Cast(NUMERIC_PRECISION AS VARCHAR(5))
ELSE ''
END + ')' AS [DataType]
FROM INFORMATION_SCHEMA.Columns
WHERE table_name = #TABLENAME
order by ordinal_Position
);
GO
once you create the function here is a sample table that I used
create table dbo.yourtable
(columna bigint,
columnb int,
columnc datetime,
columnd varchar(100),
columne char(10),
columnf bit,
columng numeric(10,2),
columnh decimal(10,2)
)
Then you can execute it as follows
select * from describe ('yourtable')
It returns the following
ColumnName IsNullable DataType
columna NO bigint(19)
columnb NO int(10)
columnc NO datetime()
columnd NO varchar(100)
columne NO char(10)
columnf NO bit()
columng NO numeric(10, 2)
columnh NO decimal(10, 2)
hope this helps someone.
As a variation of Bridge's answer (I don't yet have enough rep to comment, and didn't feel right about editing that answer), here is a version that works better for me.
SELECT column_name AS [Name],
IS_NULLABLE AS [Null?],
DATA_TYPE + CASE
WHEN CHARACTER_MAXIMUM_LENGTH IS NULL THEN ''
WHEN CHARACTER_MAXIMUM_LENGTH > 99999 THEN ''
ELSE '(' + Cast(CHARACTER_MAXIMUM_LENGTH AS VARCHAR(5)) + ')'
END AS [Type]
FROM INFORMATION_SCHEMA.Columns
WHERE table_name = 'table_name'
Notable changes:
Works for types without length. For an int column, I was seeing NULL for the type because the length was null and it wiped out the whole Type column. So don't print any length component (or parens).
Change the check for CAST length of -1 to check actual length. I was getting a syntax error because the case resulted in '*' rather than -1. Seems to make more sense to perform an arithmetic check rather than an overflow from the CAST.
Don't print length when very long (arbitrarily > 5 digits).
Just enter the below line.
exec sp_help [table_name]
I've a SQL query that queries an enormous (as in, hundreds of views/tables with hard-to-read names like CMM-CPP-FAP-ADD) database that I don't need nor want to understand. The result of this query needs to be stored in a staging table to feed a report.
I need to create the staging table, but with hundreds of views/tables to dig through to find the data types that are being represented here, I have to wonder if there's a better way to construct this table.
Can anyone advise how I would use any of the SQL Server 2008 tools to divine the source data types in my SQL 2000 database?
As a general example, I want to know from a query like:
SELECT Auth_First_Name, Auth_Last_Name, Auth_Favorite_Number
FROM Authors
Instead of the actual results, I want to know that:
Auth_First_Name is char(25)
Auth_Last_Name is char(50)
Auth_Favorite_Number is int
I'm not interested in constraints, I really just want to know the data types.
select * from information_schema.columns
could get you started.
For SQL Server 2012 and above: If you place the query into a string then you can get the result set data types like so:
DECLARE #query nvarchar(max) = 'select 12.1 / 10.1 AS [Column1]';
EXEC sp_describe_first_result_set #query, null, 0;
You could also insert the results (or top 10 results) into a temp table and get the columns from the temp table (as long as the column names are all different).
SELECT TOP 10 *
INTO #TempTable
FROM <DataSource>
Then use:
EXEC tempdb.dbo.sp_help N'#TempTable';
or
SELECT *
FROM tempdb.sys.columns
WHERE [object_id] = OBJECT_ID(N'tempdb..#TempTable');
Extrapolated from Aaron's answer here.
You can also use...
SQL_VARIANT_PROPERTY()
...in cases where you don't have direct access to the metadata (e.g. a linked server query perhaps?).
SQL_VARIANT_PROPERTY (Transact-SQL)
In SQL Server 2005 and beyond you are better off using the catalog views (sys.columns) as opposed to INFORMATION_SCHEMA. Unless portability to other platforms is important. Just keep in mind that the INFORMATION_SCHEMA views won't change and so they will progressively be lacking information on new features etc. in successive versions of SQL Server.
There MUST be en easier way to do this... Low and behold, there is...!
"sp_describe_first_result_set" is your friend!
Now I do realise the question was asked specifically for SQL Server 2000, but I was looking for a similar solution for later versions and discovered some native support in SQL to achieve this.
In SQL Server 2012 onwards cf. "sp_describe_first_result_set" - Link to BOL
I had already implemented a solution using a technique similar to #Trisped's above and ripped it out to implement the native SQL Server implementation.
In case you're not on SQL Server 2012 or Azure SQL Database yet, here's the stored proc I created for pre-2012 era databases:
CREATE PROCEDURE [fn].[GetQueryResultMetadata]
#queryText VARCHAR(MAX)
AS
BEGIN
-- SET NOCOUNT ON added to prevent extra result sets from
-- interfering with SELECT statements.
--SET NOCOUNT ON;
PRINT #queryText;
DECLARE
#sqlToExec NVARCHAR(MAX) =
'SELECT TOP 1 * INTO #QueryMetadata FROM ('
+
#queryText
+
') T;'
+ '
SELECT
C.Name [ColumnName],
TP.Name [ColumnType],
C.max_length [MaxLength],
C.[precision] [Precision],
C.[scale] [Scale],
C.[is_nullable] IsNullable
FROM
tempdb.sys.columns C
INNER JOIN
tempdb.sys.types TP
ON
TP.system_type_id = C.system_type_id
AND
-- exclude custom types
TP.system_type_id = TP.user_type_id
WHERE
[object_id] = OBJECT_ID(N''tempdb..#QueryMetadata'');
'
EXEC sp_executesql #sqlToExec
END
SELECT COLUMN_NAME,
DATA_TYPE,
CHARACTER_MAXIMUM_LENGTH
FROM information_schema.columns
WHERE TABLE_NAME = 'YOUR_TABLE_NAME'
You can use columns aliases for better looking output.
Can you get away with recreating the staging table from scratch every time the query is executed? If so you could use SELECT ... INTO syntax and let SQL Server worry about creating the table using the correct column types etc.
SELECT *
INTO your_staging_table
FROM enormous_collection_of_views_tables_etc
This will give you everything column property related.
SELECT * INTO TMP1
FROM ( SELECT TOP 1 /* rest of your query expression here */ );
SELECT o.name AS obj_name, TYPE_NAME(c.user_type_id) AS type_name, c.*
FROM sys.objects AS o
JOIN sys.columns AS c ON o.object_id = c.object_id
WHERE o.name = 'TMP1';
DROP TABLE TMP1;
select COLUMN_NAME, DATA_TYPE, CHARACTER_MAXIMUM_LENGTH
from INFORMATION_SCHEMA.COLUMNS
where TABLE_NAME='yourTable';
sp_describe_first_result_set
will help to identify the datatypes of query by analyzing datatypes of first resultset of query
https://learn.microsoft.com/en-us/sql/relational-databases/system-stored-procedures/sp-describe-first-result-set-transact-sql?view=sql-server-2017
I use a simple case statement to render results I can use in technical specification documents. This example does not contain every condition you will run into with a database, but it gives you a good template to work with.
SELECT
TABLE_NAME AS 'Table Name',
COLUMN_NAME AS 'Column Name',
CASE WHEN DATA_TYPE LIKE '%char'
THEN DATA_TYPE + '(' + CONVERT(VARCHAR, CHARACTER_MAXIMUM_LENGTH) + ')'
WHEN DATA_TYPE IN ('bit', 'int', 'smallint', 'date')
THEN DATA_TYPE
WHEN DATA_TYPE = 'datetime'
THEN DATA_TYPE + '(' + CONVERT(VARCHAR, DATETIME_PRECISION) + ')'
WHEN DATA_TYPE = 'float'
THEN DATA_TYPE
WHEN DATA_TYPE IN ('numeric', 'money')
THEN DATA_TYPE + '(' + CONVERT(VARCHAR, NUMERIC_PRECISION) + ', ' + CONVERT(VARCHAR, NUMERIC_PRECISION_RADIX) + ')'
END AS 'Data Type',
CASE WHEN IS_NULLABLE = 'NO'
THEN 'NOT NULL'
ELSE 'NULL'
END AS 'PK/LK/NOT NULL'
FROM INFORMATION_SCHEMA.COLUMNS
ORDER BY
TABLE_NAME, ORDINAL_POSITION
Checking data types.
The first way to check data types for SQL Server database is a query with the SYS schema table. The below query uses COLUMNS and TYPES tables:
SELECT C.NAME AS COLUMN_NAME,
TYPE_NAME(C.USER_TYPE_ID) AS DATA_TYPE,
C.IS_NULLABLE,
C.MAX_LENGTH,
C.PRECISION,
C.SCALE
FROM SYS.COLUMNS C
JOIN SYS.TYPES T
ON C.USER_TYPE_ID=T.USER_TYPE_ID
WHERE C.OBJECT_ID=OBJECT_ID('your_table_name');
In this way, you can find data types of columns.
This easy query return a data type bit. You can use this thecnic for other data types:
select CAST(0 AS BIT) AS OK