The dataset I am using can be found here.
I am using SSIS to upload the dataset into MS SQL Server.
I uploaded everything as text, and am trying to create a working table with proper data types by inserting values from the raw table.
CREATE TABLE [WRK_demographics]
(
[RowNumber] INT IDENTITY(1,1)
,[DBN] VARCHAR(10)
,[Name] VARCHAR(1000)
,[schoolyear] VARCHAR(100)
,[fl_percent] FLOAT
,[frl_percent] FLOAT
,[total_enrollment] INT
,[grade9] INT
,[grade10] INT
,[grade11] INT
,[grade12] INT
,[ell_num] INT
,[ell_percent] FLOAT
,[sped_num] INT
,[sped_percent] FLOAT
,[ctt_num] INT
,[selfcontained_num] INT
,[asian_num] INT
,[asian_per] FLOAT
,[black_num] INT
,[black_per] FLOAT
,[hispanic_num] INT
,[hispanic_per] FLOAT
,[white_num] INT
,[white_per] FLOAT
,[male_num] INT
,[male_per] FLOAT
,[female_num] INT
,[female_per] FLOAT
)
INSERT INTO [WRK_demographics]
(
[DBN]
,[Name]
,[schoolyear]
,[fl_percent]
,[frl_percent]
,[total_enrollment]
,[grade9]
,[grade10]
,[grade11]
,[grade12]
,[ell_num]
,[ell_percent]
,[sped_num]
,[sped_percent]
,[ctt_num]
,[selfcontained_num]
,[asian_num]
,[asian_per]
,[black_num]
,[black_per]
,[hispanic_num]
,[hispanic_per]
,[white_num]
,[white_per]
,[male_num]
,[male_per]
,[female_num]
,[female_per]
)
SELECT
[DBN]
,[Name]
,[schoolyear]
,[fl_percent]
,[frl_percent]
,[total_enrollment]
,[grade9]
,[grade10]
,[grade11]
,[grade12]
,[ell_num]
,[ell_percent]
,[sped_num]
,[sped_percent]
,[ctt_num]
,[selfcontained_num]
,[asian_num]
,[asian_per]
,[black_num]
,[black_per]
,[hispanic_num]
,[hispanic_per]
,[white_num]
,[white_per]
,[male_num]
,[male_per]
,[female_num]
,[female_per]
FROM [RAW_demographics_20170706]
However, the issue I am having is that for cells where there is no value, there is text instead of a null cell. So, implicit conversion cannot convert the data to int/float. Is there a way I can modify the SELECT statement to update the values of non-numeric cells to NULL? If I didn't have so many columns I would update each one in the raw table like this before inserting into the working table:
UPDATE [RAW_demographics_20170706]
SET [fl_percent] = NULL
WHERE ISNUMERIC([fl_percent]) <> 1
I'm wondering if there is a more efficient path to take.
You'd want to use a case statement since your error should be coming from empty strings not an actual NULL
SELECT
[DBN]
,[Name]
,[schoolyear]
,case when [fl_percent] = '' then 0.0 else fl_percent end
,case when [frl_percent] = '' then 0.0 else flr_percent end
...
...
,case when [grade12] = '' then 0 else grade12 end
...
FROM [RAW_demographics_20170706]
I also wouldn't use ISNUMERIC for this situation. It will cause problems as it returns true for an array of cases other than an int or float.
These return true, but would fail for conversions:
select isnumeric('$')
select isnumeric('1e4')
etc...
That's what TRY_CAST is for: "Returns a value cast to the specified data type if the cast succeeds; otherwise, returns null."
Related
I have an issue where I am trying to add a leading 0 to run an output.
SELECT
CASE
WHEN LEN(t.trans_time) = 5
THEN CONCAT(0, [trans_time])
ELSE T.[trans_time]
END AS [TransactionTime]
,RIGHT(CONCAT(0,trans_time),6) AS trans_time
,LEN(T.trans_Time)
,t.trans_time
Why does the case statement not return the leading 0 whereas using:
,RIGHT(CONCAT(0,trans_time),6) AS trans_time
Works no problem.
Case expression return only one type, whereas concat() would return different type & i am assuming trans_time has INT type.
So, you would need to do type conversations :
SELECT (CASE WHEN LEN(t.trans_time) = 5
THEN CONCAT(0, [trans_time])
ELSE CAST(T.[trans_time] AS VARCHAR(255))
END) AS [TransactionTime],
. . .
Another way to do this is to use the format function, wich is available from sql server 2012.
It not only makes the code more readable but will also perform better.
declare #t table (id int)
insert into #t values (90113), (90204), (90207), (90235), (90302), (90318), (90324)
select format(id, '000000') as TransactionTime from #t
this will return
TransactionTime
---------------
090113
090204
090207
090235
090302
090318
090324
Is there any possible way to improve the below query:
DECLARE #radiusInMeters FLOAT = 400;
DECLARE #dgeog geography = geography::Point(given_latitude, given_longitude, 4326).STBuffer(#radiusInMeters);
select [fdx].latitude, [fdx].longitude
from [dbo].[fdx]
where #dgeog.STIntersects(geography::STGeomFromText('POINT(' + convert(varchar(20), [fdx].longitude) + ' ' + convert(varchar(20), [fdx].latitude) + ')', 4326)
) = 1
kcung and Hasan BINBOGA are correct, you need a spatial index.
Look at your query:
#dgeog.STIntersects(xxxx) = 1
This requires [xxxx] to be a geography data type. In order for [xxxx] to be a geography data type, the STGeomFromText function must be applied to the row. And because this is the only part of your WHERE clause, the function must be applied to all rows.
If the table fdx is particularly large, this means that the CLR function will have to be applied over and over again. This is not (in SQL-Server terms) a fast process.
Try this, if you can:
ALTER dbo.fdx ADD Point AS (GEOGRAPHY::Point(Latitude, Longitude, 4326)) PERSISTED
GO
CREATE SPATIAL INDEX SIndex_FDX ON dbo.fdx (Point)
USING GEOGRAPHY_GRID
WITH (
GRIDS = (LEVEL_1 = HIGH,LEVEL_2 = HIGH,LEVEL_3 = HIGH,LEVEL_4 = HIGH),
CELLS_PER_OBJECT = 1
)
GO
DECLARE #Latitude DECIMAL(15,10) = 0
DECLARE #Longitude DECIMAL(15,10) = 0
DECLARE #Radius FLOAT = 400
DECLARE #g GEOGRAPHY = GEOGRAPHY::Point(#Latitude, #Longitude, 4326).STBuffer(#Radius)
SELECT * FROM dbo.fdx WHERE Point.STIntersects(#g) = 1
A note: You should convert your lat/long pairs into decimals before using them to compute the geography column. There is an implicit conversion from float to decimal to string when you use a float as an input that will trim your coordinates down to 4 decimal places. If you explicitly convert first, that will not be an issue.
Also, if you have any null lat/long values in dbo.fdx, you need to filter them in the WHERE clause as a null value will cause your spatial index not to work properly.
You can create spatial index :
https://msdn.microsoft.com/en-us/library/bb934196.aspx
I'm looking to parse a sql column result into separate columns. Here is an example of the column...
Detail - Column name
'TaxID changed from "111" to "333". Address1 changed from "542 Test St." to "333 Test St". State changed from "FL" to "DF". Zip changed from "11111" to "22222". Country changed from "US" to "MX". CurrencyCode changed from "usd" to "mxn". RFC Number changed from "" to "test". WarehouseID changed from "6" to "1". '
I need to take the old TAXID, new TAXID, old country, and new country and put them in separate columns.
The Detail column will always have TAXID and Country, however the challenging part is that they don't always have the rest of data that I listed above. Sometimes it will contain city and other times it won't. This means the order is always different.
I would create a tsql proc, use a case statement.
Do a count of the double quotes. If there are 8 oairs, you know that you old and new values, only 4 pairs you only have new values.
Then using the double quotes as indexes for your substring, you can put the vales into the table.
Good luck!
I was able to come up with something that worked.
In case anyone else gets a situation like this again perhaps posting my code will help.
DECLARE #document varchar(350);
set #document = 'TaxID changed from "111" to "222"'
declare #FIRSTQUOTE int
declare #SECONDQUOTE int
declare #OLDTAXID nvarchar(40)
declare #firstlength int
declare #ThirdQuote int
declare #FourthQuote int
declare #secondlength int
declare #NewTAXID nvarchar(40)
declare #oneplussecondquote int
declare #oneplusthirdquote int
select #FirstQuote = CHARINDEX('"',#document)
set #FIRSTQUOTE = #FIRSTQUOTE + 1
select #SECONDQUOTE = CHARINDEX('"',#document,#FIRSTQUOTE)
set #firstlength = #SECONDQUOTE - #FIRSTQUOTE
select #OLDTAXID = SUBSTRING(#document,#FIRSTQUOTE,#firstlength)
set #oneplussecondquote = #SECONDQUOTE + 1
select #ThirdQuote = CHARINDEX('"',#document,#oneplussecondquote)
set #oneplusthirdquote = #ThirdQuote + 1
select #FourthQuote = CHARINDEX('"',#document,#oneplusthirdquote)
select #secondlength = #FourthQuote - #oneplusthirdquote
select #NewTAXID = SUBSTRING(#document,#oneplusthirdquote,#secondlength)
You can switch out the string for this: 'Country changed from "US" to "MX"'
And it would grab the old country and new country
I'm hoping that what I have paints a clear enough picture of what I am trying to accomplish:
SELECT [Date]
,[ChargeCode]
,[ChargeDescription]
,[HHY_Qty]
,[PatPrice]
, IF ISNUMERIC(HHY_Qty) AND ISNUMERIC(PatPrice)
BEGIN
CAST(HHY_Qty AS INT) * CAST(PatPrice AS INT) AS ExtAmt
END
ELSE
0 AS ExtAmt
END
FROM [dbo].[ChargeDetails]
WHERE PatientNumber = '1271'
HHY_Qty and PatPrice are both VARCHAR types in a MSSQL database. They were created with a BULK INSERT from a very very very dirty CSV from an AS400 export. Here, I am trying to do some multiplication IF the fields are numeric values, otherwise ExtAmt should be 0. Is that possible? If not,, is there a workaround?
Use a CASE statement:
CASE WHEN ISNUMERIC(HHY_Qty) = 1 AND ISNUMERIC(PatPrice) = 1 THEN CAST(HHY_Qty AS INT) * CAST(PatPrice AS INT) ELSE 0 END AS ExtAmt
I have imported data into a database from an excel file and some of the columns contain values are like (392.03), (2.25), (65.00). Actually these values should be -ve values can you guys help me to convert these into -392.03,-2.25,-65.00
DECLARE #NumValue DEC (9,3) = 0
DECLARE #StrValue varchar(15) = '(392.03)'
IF LEFT(#StrValue,1)='(' and RIGHT(#StrValue,1)=')'
BEGIN
SET #StrValue = SUBSTRING(#StrValue,2,LEN(#StrValue)-2)
SET #NumValue = convert(DEC (9,3), #StrValue)
END
SELECT #NumValue
Try this in the derived column
SUBSTRING([Paid Amount],1,1) == "(" ? REPLACE(REPLACE([Paid Amount],"(","-"),")","") : [Paid Amount]
Then use a Data conversion step to convert the value yo Float DT_R4