Please have a look at the code below and explain to me why there is a deviance in the final results. Note that the difference is the introduction of the brackets in the second calculation. Thanks!
Code:
DECLARE #A decimal(38,19) = 7958011.98
DECLARE #B decimal(38,19) = 10409029441
DECLARE #C decimal(38,19) = 10000000000
DECLARE #Z1 decimal(38,19)
DECLARE #Z2 decimal(38,19)
SET #Z1 = #A * #B / #C
SET #Z2 = #A * (#B / #C)
SELECT #Z1 AS [Correct],
#Z2 AS [Wrong]
Results:
Correct = 8283518.0991650000000000000
Wrong = 8283510.5860060000000000000
The intermediate datatypes are different because of this MSDN article
That is, (#B / #C) evaluated first, follows rules like this. The intermediate datatype then affects the multiplication by #A
You can see the intermediate and final types here (before assigning to a decimal(38,19) type
SELECT
#A * #B, -- decimal (x, 6)
#A * #B / #C, -- decimal (x, 6)
(#B / #C), -- decimal (x, 6)
#A * (#B / #C) -- decimal (x, 6)
So, instead of 1.0409029441 you get 1.040902 for your 2nd math
Note, your 1st is wrong too. It is actually 8283518.099165070318
Related
I have 2 registers transfer from PLC (modbus) to sql server, each 16 bit. I want to combine this 2 integers, and to convert to a Real sql datatype
declare #int1, #int2 integer
declare #result real
Select #int1 = 18196
, #int2 = 28800
Select #result = #int1 * 65536 + #int2 //-- this result is not wanted
I need the same result as in my Plc, the result is 38000.5
How do I convert to the real data type in sql???
My head hurts, of searching the net, thanks for help
The other way around is
select #result = 38000.5
Select convert(integer, substring(convert(varbinary, #result),1,2) //18196
Select convert(integer, substring(convert(varbinary, #result),3,2) //28800
declare #result real
declare #first int, #second int
set #result=38000.5
set #first = convert(integer, substring(convert(varbinary, #result),1,2)) --18196
set #second = convert(integer, substring(convert(varbinary, #result),3,2)) --28800
print #first
print #second
declare #binaryfloat varbinary(4)=convert(varbinary(2),#first)+convert(varbinary(2),#second);
SELECT SIGN(CAST(#BinaryFloat AS INT))
* (1.0 + (CAST(#BinaryFloat AS INT) & 0x007FFFFF) * POWER(CAST(2 AS REAL), -23))
* POWER(CAST(2 AS REAL), (CAST(#BinaryFloat AS INT) & 0x7f800000) / 0x00800000 - 127)
But as #honeybadger wrote, why not use normal types?
While working with str() function I found that in some cases it rounds wrong while round() function works as expected. Please look at the example:
declare #v decimal(18,2) = 29.95
select str(#v, 18, 1)
--29.9
select round(#v, 1)
--30.00
set #v = 39.95
select str(#v, 18, 1)
--40.00
select round(#v, 1)
--40.00
Can anyone explain why it happens?
EDIT1:
I tested different workarounds with the following base code:
declare #v decimal(18,2) = 9.95
declare #r varchar(100)
declare #c int = 1000000
declare #ms int
declare #dt datetime2
set #dt = sysdatetime()
while #c > 0
begin
set #r = --different roundings
set #c = #c - 1
end
set #ms = DATEDIFF(ms, #dt, sysdatetime())
select #ms, #r
Option 1 (the original one, rounds wrongly in some cases):
str(#v, 18, 1)
Option 2 (slightly modified but rounds correctly):
str(round(#v, 1), 18, 1)
Option 3 (double conversion and rounding):
convert(varchar(20), convert(decimal(18,1), round(#v, 1)))
Option 4 (only double conversion):
convert(varchar(20), convert(decimal(18,1), #v))
Results:
Option 1 and 2 are roughly 2 times slower than the last two but the result is right-justified. The fastest is Option 4.
The parameter to str() is a float so your decimal value is implicitly converted to a float(53) that is then converted to a string. So you see a floating point rounding error.
Do a slight modification to your query and you can see what is happening in the actual execution plan.
declare #v decimal(18,2) = 29.95
select top(1) str(#v, 18, 1)
<ScalarOperator ScalarString="str(CONVERT_IMPLICIT(float(53),[#v],0),(18),(1))">
Syntax of STR(); STR ( float_expression [ , length [ , decimal ] ] ) clearly says that the number is a float_expression. Therefore whatever the number you give it will be first converted to a FLOAT(n) where default value of n = 53.
So
SELECT STR(4.65,5,1), SELECT STR(3.65,5,1)
Equal to:
SELECT STR(CAST(4.65 AS FLOAT(53)),5,1) , STR(CAST(3.65 AS FLOAT(53)),5,1)
If you specify n, say n = 4 it will give the answer you are expecting (ie; 4.7 and 3.7)
SELECT STR(CAST(4.65 AS FLOAT(4)),5,1) , STR(CAST(3.65 AS FLOAT(4)),5,1)
--4.7, 3.7
Suppose I have two number, #n1, #n2, I want to get bigger one in one simple expression like Max(#n1,#n2). How to write the expression for T-SQL?
DECLARE
#n1 INT = 2,
#n2 INT = 3
SELECT MAX(n) FROM (VALUES(#n1), (#n2)) t(n)
CASE WHEN #n1 > #n2 THEN #n1 ELSE #n2 END
DECLARE #a int = 45, #b int = 40;
SELECT IIF ( #a > #b, #a, #b ) AS Result;
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Truncate (not round) decimal places in SQL Server
Can't figure this out. I need 1 decimal place returned while SQL is rounding to whole numbers.
I read that integers divided by integers give integers in SQL, but I need the one truncated decimal place for value of output in the temp table.
I don't mind if 35.0 comes back as 35 but 35.17 should come back as 35.1. Sorry just edited. Need to truncate the last number, not round up.
create table #blah(output decimal(9,1))
DECLARE #a money
DECLARE #b money
DECLARE #intinterval decimal(9,1)
SET #a = 5
SET #b = 2
SET #intinterval = (#b / 1000.0) * (86400.0 / #a)
INSERT INTO #blah (output) VALUES (#intinterval)
SELECT * from #blah
drop table #blah
The above equation should give (2 / 1000) * (86400 / 5) = (0.002 * 17280) = 34.56
The 34.56 should truncate to 34.5
SET #intinterval = cast(10 * (#b / 1000.0) * (86400.0 / #a) as int) / 10.0
or
SET #intinterval = cast(#b * 864.0 / #a as int) / 10.0
What about Round((#b / 1000.0) * (86400.0 / #a), 1, 1), where last 1 saying to truncate instead of round.
try this with no special functions...
if
a = 5 then output = 34.5 (34.56)
a = 7 output = 24.6 (24.69)
a = 11 output = 15.7 (15.71)
create table #blah(output decimal(9,1))
DECLARE #a money
DECLARE #b money
DECLARE #intinterval decimal(9,2)
declare #rounded decimal(9,1)
declare #diff decimal(9,2)
declare #finalvalue decimal(9,1)
SET #a = 5
SET #b = 2
SET #intinterval = (#b / 1000.0) * (86400.0 / #a)
set #rounded = #intinterval -- gets the rounded value
set #diff = #intinterval - #rounded -- gets the difference whether to round off or not
if #diff >= 0 -- if differnce is >= 0 then get the rounded value .. eg. 34.31 = 34.3
set #finalvalue = #rounded
else -- else subtract 0.1 and get the rounded value e.g. 34.56 - 0.1 = 34.46 -> rounded value of 34.5
set #finalvalue = #intinterval - 0.1
INSERT INTO #blah (output) VALUES (#finalvalue )
SELECT * from #blah
drop table #blah
I have the following SQL query that comes out at 700
declare #a decimal(10,0)
declare #b decimal(10,0)
set #a = 100 - 2
set #a = #a / 14
set #a = #a * 100
set #a = ((100 - 2) / 14) * 100
select #a
What i am looking for it do is return 85.714285 etc...
Not quite sure where i going wrong.
Thanks
Your declaration is wrong:
declare #a decimal(10,0)
declare #b decimal(10,0)
You have specified the length, however not the number of decimals, which is done by using the second value.
700 is the correct result.
Please check again!
((100 - 2) / 14) * 100 = 700
You calculated that
100 - 2 / 14 * 100 = 85,...
You want to use this sql query.
set #a = 100 - 2.0 / 14 * 100
Add a .0 to the end of your last line, since if you use all integers SQL will implicitly cast the result as an int.
set #a = ((100 - 2) / 14) * 100.0
change your declarations to include decimal places:
declare #a decimal(10,5)
declare #b decimal(10,5)
set #a = 100 - 2
set #a = #a / 14
set #a = #a * 100
set #a = ((100 - 2) / 14) * 100
select #a
Since your declaration is set to 10,0 then you will get no decimal places.
if you want the answer to be 85.714... then you need to change your SQL to:
declare #a decimal(10,5)
set #a = 100.0 - 2.0
set #a = #a / 14.0
set #a = #a * 100.0
set #a = 100.0 - ((2.0 / 14.0) * 100.0)
select #a
To get the result you are looking for you need to add a . to the other values (2, 14, etc) and you will get the value you want, you also need to make sure that your parentheses are in the correct location.
Results - 85.71430