I have the following set of calculations in excel that I want to be able to use in a stored procedure.
Excel
CellA: 45/448.2 = 0.100401606425703
CellB: 1-CellA = 0.899598393574297
CellC: 1-CellB = 0.100401606425703
CellD: CellC * 448.2 = 45.000000000000000
In SQL I am doing the following:
declare #a decimal(18,15) = 45/448.2
declare #b decimal(18,15) = 1-#a
declare #c decimal(18,15) = 1-#b
declare #d decimal(18,15) = #c * 448.2
I have also tried running the calculation in one line
declare #e decimal(18,15) = (1-(1-(45/448.2)))*448.2
when I return the values SQL gives me the following:
#a: 0.100401000000000
#b: 0.899599000000000
#c: 0.100401000000000
#d: 44.999728200000000
#e: 44.999728200000000
I've tried adjusting the precision of the decimals in SQL but I nothing makes a difference, it only returns the first 6 digits of the decimal.
Does Excel do any optimization when running the formula?
Any ideas?
Even just your first line is enough to show the problem:
declare #a decimal(18,15) = 45/448.2
print #a
gives
---------------------------------------
0.100401000000000
This is because of data types. When you say
448.2
it is (per the documentation) interpreted as a constant of type decimal, and also per the documentation,
In Transact-SQL statements, a constant with a decimal point is
automatically converted into a numeric data value, using the minimum
precision and scale necessary. For example, the constant 12.345 is
converted into a numeric value with a precision of 5 and a scale of 3.
So 448.2 is decimal(4,3). 45 is integer, which when combined with a decimal is treated as having precision of 10 and scale 0 . When we divide, the rules say
Operation Result precision Result scale
e1 / e2 p1 - s1 + s2 + max(6, s1 + p2 + 1) max(6, s1 + p2 + 1)
which in this case gives a result precision of 10 - 3 + 0 + max(6, 0 + 3 + 1) and scale of max(6, 0 + 3 + 1), which comes out to 13 and 6.
That result scale of 6 is why the result only has those six decimal places.
The way to fix it is to get your operands into an appropriate type before acting on them; for example, here are two ways:
Force a number to be treated as floating-point:
declare #a decimal(18,15) = 45/448.2e0
select #a
---------------------------------------
0.100401606425703
Explicitly supply a decimal scale:
declare #a decimal(18,15) = 45/cast(448.2 as decimal(18,10))
select #a
---------------------------------------
0.100401606425703
Related
I have been calculating different integer percentages with different numbers but each time I get floor rounded number. select 13*100/60 gives me 21 and the actual number is 21.66 which using a round function should give us 22 but it can only give me 21 for all different decimal numbers.
I am using SQL 2017. please help
This is due to the fact that you are dividing ints and not floating-point numbers. Integer division returns an integer.
Try the following instead (noting the .0 on the end of the 60):
SELECT 13 * 100 / 60.0
Making one of the components a floating-point number will automatically output the result as a floating-point number.
Output:
21.666666
Incidentally, if you are working with variables and one of them is a FLOAT, it will automatically produce the output you expect:
DECLARE #A FLOAT
DECLARE #B INT
DECLARE #C INT
SET #A = 13
SET #B = 100
SET #C = 60
SELECT #A * #B / #C
Output:
21.6666666666667
This post has the following code:
DECLARE #A DECIMAL(3, 0), #B DECIMAL(18, 0), #F FLOAT
SET #A = 3
SET #B = 3
SET #F = 3
SELECT 1 / #A * 3.0, 1 / #B * 3.0, 1 / #F * 3.0
SELECT 1 / #A * 3 , 1 / #B * 3 , 1 / #F * 3
Using float, the expression evaluates to 1. Using Decimal, the expression evaluates to some collection of 9s after the decimal point. Why does float yield the more accurate answer in this case? I thought that Decimal is more accurate / exact per Difference between numeric, float and decimal in SQL Server and Use Float or Decimal for Accounting Application Dollar Amount?
The decimal values that you have declared are fixed width, and there are no points after the decimal place. This affects the calculations.
SQL Server has a rather complex formula for how to calculate the precision of arithmetical expressions containing decimal numbers. The details are in the documentation. You also need to take into account that numeric constants are in decimal format, rather than numeric.
Also, in the end, you need to convert back to a decimal format with the precision that you want. In that case, you might discover that float and decimal are equivalent.
As far as i know according to mathematics rounding should work as below when rounding number is 5.
2.435 => 2.44 (Round Up, if rounding to digit(3) is odd number)
2.445 => 2.44 (Round Down, if rounding to digit(4) is even number)
if we do summation all fine,
2.435 + 2.445 = 4.88
2.44 + 2.44 = 4.88
I'm pretty sure in .Net also rounding works like this.
But in SQL server, 5 is always rounding up which is not correct according to maths.
SELECT round(2.345, 2) = 2.35
SELECT round(2.335, 2) => 2.34
this results to 1 cent discrepancies in summation of rounded values.
2.345 + 2.335 = 4.68
2.35 + 2.34 = 4.69 => which is not correct
I have tried this with decimal and money data types.
Am i doing something wrong? Is there a work around for this?
If you do want to use banker's rounding in SQL Server...
CREATE FUNCTION BankersRounding(#value decimal(36,11), #significantDigits INT)
RETURNS MONEY
AS
BEGIN
-- if value = 12.345 and signficantDigits = 2...
-- base = 1000
declare #base int = power(10, #significantDigits + 1)
-- roundingValue = 12345
declare #roundingValue decimal(36,11) = floor(abs(#value) * #base)
-- roundingDigit = 5
declare #roundingDigit int = #roundingValue % 10
-- significantValue = 1234
declare #significantValue decimal(36,11) = floor(#roundingValue / 10)
-- lastSignificantDigit = 4
declare #lastSignificantDigit int = #significantValue % 10
-- awayFromZero = 12.35
declare #awayFromZero money = (#significantValue + 1) / (#base / 10)
-- towardsZero = 12.34
declare #towardsZero money = #significantValue / (#base / 10)
-- negative values handled slightly different
if #value < 0
begin
-- awayFromZero = -12.35
set #awayFromZero = ((-1 * #significantValue) - 1) / (#base / 10)
-- towardsZero = -12.34
set #towardsZero = (-1 * #significantValue) / (#base / 10)
end
-- default to towards zero (i.e. assume thousandths digit is 0-4)
declare #rv money = #towardsZero
if #roundingDigit > 5
set #rv = #awayFromZero -- 5-9 goes away from 0
else if #roundingDigit = 5
begin
-- 5 goes to nearest even number (towards zero if even, away from zero if odd)
set #rv = case when #lastSignificantDigit % 2 = 0 then #towardsZero else #awayFromZero end
end
return #rv
end
You're looking for Banker's Rounding - which is the default rounding in C# but is not how SQL Server ROUND() works.
See Why does TSQL on Sql Server 2000 round decimals inconsistently? as well as http://blogs.lessthandot.com/index.php/DataMgmt/DataDesign/sql-server-rounding-methods and http://www.chrispoulter.com/blog/post/rounding-decimals-using-net-and-t-sql
Mathematically rounding up at 5 is correct, and also the most commonly used type of rounding in basic mathematics. Other types of rounding are also valid, but are not basic mathematics, but more often used in certain areas due to 0.5 often being a dispute number.
What you call mathematically rounding is actually bankers rounding, which is the type of rounding used in the finance business.
Is there a way in T-SQL to convert a TINYINT to VARCHAR with custom number formatting?
For instance, my TINYINT has a value of 3 and I want to convert it to a VARCH of 03, so that it always shows a 2 digit number.
I don't see this ability in the CONVERT function.
RIGHT('00' + CONVERT(VARCHAR, MyNumber), 2)
Be warned that this will cripple numbers > 99. You might want to factor in that possibility.
Use the RIGHT function...
e.g.
DECLARE #testnum TINYINT
SET #testnum = 3
PRINT RIGHT('00' + CONVERT(VARCHAR(2), #testnum), 2)
You can try this
DECLARE #Table TABLE(
Val INT
)
INSERT INTO #Table SELECT 3
INSERT INTO #Table SELECT 30
DECLARE #NumberPrefix INT
SET #NumberPrefix = 2
SELECT REPLICATE('0', #NumberPrefix - LEN(Val)) + CAST(Val AS VARCHAR(10))
FROM #Table
What is the value range? Is it 0 through 10? If so, then try:
SELECT REPLICATE('0',2-LEN(#t)) + CAST(#t AS VARCHAR)
That handles 0 through 9 as well as 10 through 99.
Now, tinyint can go up to the value of 255. If you want to handle > 99 through 255, then try this solution:
declare #t TINYINT
set #t =233
SELECT ISNULL(REPLICATE('0',2-LEN(#t)),'') + CAST(#t AS VARCHAR)
To understand the solution, the expression to the left of the + calculates the number of zeros to prefix to the string.
In case of the value 3, the length is 1. 2 - 1 is 1. REPLICATE Adds one zero.
In case of the value 10, the length is 2. 2 - 2 is 0. REPLICATE Adds nothing.
In the case of the value 100, the length is -1 which produces a NULL. However, the null value is handled and set to an empty string.
Now if you decide that because tinyint can contain up to 255 and you want your formatting as three characters, just change the 2-LEN to 3-LEN in the left expression and you're set.
declare #t tinyint
set #t =3
select right(replicate('0', 2) + cast(#t as varchar),2)
Ditto: on the cripping effect for numbers > 99
If you want to cater for 1-255 then you could use
select right(replicate('0', 2) + cast(#t as varchar),3)
But this would give you 001, 010, 100 etc
Here's an alternative following the last answer
declare #t tinyint,#v tinyint
set #t=23
set #v=232
Select replace(str(#t,4),' ','0'),replace(str(#t,5),' ','0')
This will work on any number and by varying the length of the str() function you can stipulate how many leading zeros you require. Provided of course that your string length is always >= maximum number of digits your number type can hold.
Correción: 3-LEN
declare #t TINYINT
set #t =233
SELECT ISNULL(REPLICATE('0',3-LEN(#t)),'') + CAST(#t AS VARCHAR)
Had the same problem with a zipcode field. Some folks sent me an excel file with zips, but they were formatted as #'s. Had to convert them to strings as well as prepend leading 0's to them if they were < 5 len ...
declare #int tinyint
set #int = 25
declare #len tinyint
set #len = 3
select right(replicate('0', #len) + cast(#int as varchar(255)), #len)
You just alter the #len to get what you want. As formatted, you'll get...
001
002
...
010
011
...
255
Ideally you'd "varchar(#len)", too, but that blows up the SQL compile. Have to toss an actual # into it instead of a var.
I would like to be able to round a number to n significant figures in SQL. So:
123.456 rounded to 2sf would give 120
0.00123 rounded to 2sf would give 0.0012
I am aware of the ROUND() function, which rounds to n decimal places rather than significant figures.
select round(#number,#sf-1- floor(log10(abs(#number)))) should do the trick !
Successfully tested on your two examples.
Edit : Calling this function on #number=0 won't work. You should add a test for this before using this code.
create function sfround(#number float, #sf int) returns float as
begin
declare #r float
select #r = case when #number = 0 then 0 else round(#number ,#sf -1-floor(log10(abs(#number )))) end
return (#r)
end
Adapted the most popular answer by Brann to MySQL for those who come looking like me.
CREATE FUNCTION `sfround`(num FLOAT, sf INT) # creates the function
RETURNS float # defines output type
DETERMINISTIC # given input, will return same output
BEGIN
DECLARE r FLOAT; # make a variable called r, defined as a float
IF( num IS NULL OR num = 0) THEN # ensure the number exists, and isn't 0
SET r = num; # if it is; leave alone
ELSE
SET r = ROUND(num, sf - 1 - FLOOR(LOG10(ABS(num))));
/* see below*/
END IF;
RETURN (r);
END
/* Felt too long to put in comment */
ROUND(num, sf - 1 - FLOOR(LOG10(ABS(num))))
The part that does the work - uses ROUND function on the number as normal, but the length to be rounded to is calculated
ABS ensures positive
LOG10 gets the number of digits greater than 0 in the number
FLOOR gets the largest integer smaller than the resultant number
So always rounds down and gives an integer
sf - 1 - FLOOR(...) gives a negative number
works because ROUND(num, -ve num) rounds to the left of the decimal point
For just a one off, ROUND(123.456, -1) and ROUND(0.00123,4)
return the requested answers ((120, 0.0012)
I think I've managed it.
CREATE FUNCTION RoundSigFig(#Number float, #Figures int)
RETURNS float
AS
BEGIN
DECLARE #Answer float;
SET #Answer = (
SELECT
CASE WHEN intPower IS NULL THEN 0
ELSE FLOOR(fltNumber * POWER(CAST(10 AS float), intPower) + 0.5)
* POWER(CAST(10 AS float), -intPower)
END AS ans
FROM (
SELECT
#Number AS fltNumber,
CASE WHEN #Number > 0
THEN -((CEILING(LOG10(#Number)) - #Figures))
WHEN #Number < 0
THEN -((FLOOR(LOG10(#Number)) - #Figures))
ELSE NULL END AS intPower
) t
);
RETURN #Answer;
END
You could divide by 100 before rounding and then multiplying by 100...