Does anyone know why, using SQLServer 2005
SELECT CONVERT(DECIMAL(30,15),146804871.212533)/CONVERT(DECIMAL (38,9),12499999.9999)
gives me 11.74438969709659,
but when I increase the decimal places on the denominator to 15, I get a less accurate answer:
SELECT CONVERT(DECIMAL(30,15),146804871.212533)/CONVERT(DECIMAL (38,15),12499999.9999)
give me 11.74438969
For multiplication we simply add the number of decimal places in each argument together (using pen and paper) to work out output dec places.
But division just blows your head apart. I'm off to lie down now.
In SQL terms though, it's exactly as expected.
--Precision = p1 - s1 + s2 + max(6, s1 + p2 + 1)
--Scale = max(6, s1 + p2 + 1)
--Scale = 15 + 38 + 1 = 54
--Precision = 30 - 15 + 9 + 54 = 72
--Max P = 38, P & S are linked, so (72,54) -> (38,20)
--So, we have 38,20 output (but we don use 20 d.p. for this sum) = 11.74438969709659
SELECT CONVERT(DECIMAL(30,15),146804871.212533)/CONVERT(DECIMAL (38,9),12499999.9999)
--Scale = 15 + 38 + 1 = 54
--Precision = 30 - 15 + 15 + 54 = 84
--Max P = 38, P & S are linked, so (84,54) -> (38,8)
--So, we have 38,8 output = 11.74438969
SELECT CONVERT(DECIMAL(30,15),146804871.212533)/CONVERT(DECIMAL (38,15),12499999.9999)
You can do the same math if follow this rule too, if you treat each number pair as
146804871.212533000000000 and 12499999.999900000
146804871.212533000000000 and 12499999.999900000000000
To put it shortly, use DECIMAL(25,13) and you'll be fine with all calculations - you'll get precision right as declared: 12 digits before decimal dot, and 13 decimal digits after.
Rule is: p+s must equal 38 and you will be on safe side!
Why is this?
Because of very bad implementation of arithmetic in SQL Server!
Until they fix it, follow that rule.
I've noticed that if you cast the dividing value to float, it gives you the correct answer, i.e.:
select 49/30 (result = 1)
would become:
select 49/cast(30 as float) (result = 1.63333333333333)
We were puzzling over the magic transition,
P & S are linked, so:
(72,54) -> (38,29)
(84,54) -> (38,8)
Assuming (38,29) is a typo and should be (38,20), the following is the math:
i. 72 - 38 = 34,
ii. 54 - 34 = 20
i. 84 - 38 = 46,
ii. 54 - 46 = 8
And this is the reasoning:
i. Output precision less max precision is the digits we're going to throw away.
ii. Then output scale less what we're going to throw away gives us... remaining digits in the output scale.
Hope this helps anyone else trying to make sense of this.
Convert the expression not the arguments.
select CONVERT(DECIMAL(38,36),146804871.212533 / 12499999.9999)
Using the following may help:
SELECT COL1 * 1.0 / COL2
Related
In a table having an integer primary key of indexRow in which the last two digits are currently 55, I'd like to change that to 50 but only if the column added is an integer value 55 and the indexRow ends in 55. I'm using SQLite.
I tested it as follows. Would you please tell me if this is the correct approach (if there is a better method) because I'd like to use it to run an update on the table?
Of course, I'll do it within a transaction and test before committing; but wanted to ask. I expected to have to use some math to determine which indexRows ended in 55, but converting to string seems quite easy.
select indexRow, indexRow-5, substring(format('%s', indexRow),-2)
from newSL
where added=55
and substring(format('%s', indexRow),-2)='55'
limit 10;
indexRow indexRow-5 substring(format('%s', indexRow),-2)
----------- ----------- ------------------------------------
10080171455 10080171450 55
10130031255 10130031250 55
10140021655 10140021650 55
10140080955 10140080950 55
10240330155 10240330150 55
10250230555 10250230550 55
10270031155 10270031150 55
10270290355 10270290350 55
10300110355 10300110350 55
10300110455 10300110450 55
Yes, use the modulo operator, %. In the expression x % y, the result is the remainder of dividing x by y. Therefore, 4173 % 100 = 73.
Note that % is a math operator, just like * for multiplication and / for division, and is not related to using the % in the format function.
I have to check how many hundreds are there in a number and translate that number to letters. For example the number 700. I have done the following code:
DATA(lv_dmbtr) = ZDS_FG-DMBTR. //Declared local variable of type DMBTR, thus DMBTR=700.
lv_dmbtr = ZDS_FG-DMBTR MOD 100. //Finding how many times 700 is in 100 via MOD and putting the value in lv_dmbtr.
IF lv_dmbtr LE 9. //The value is less or equal than 9(if larger means that the DMBTR is larger than hundreds,
e.g. 8000)
lv_hundred = lv_dmbtr / 100. // Divide the 700 with 100, taking the number 7.
lv_hundred_check = lv_hundred MOD 1. // Then taking the value of 7 into the new variable, done in case the
lv_hundred is a decimal value, e.g. 7.32.
IF lv_hundred_check > 0.
CALL FUNCTION 'SPELL_AMOUNT'
EXPORTING
amount = lv_hundred_check
* CURRENCY = ' '
* FILLER = ' '
LANGUAGE = SY-LANGU
IMPORTING
in_words = lv_hundred_string // the value is put in the new string
EXCEPTIONS
not_found = 1
too_large = 2
OTHERS = 3.
ENDIF.
Now when I debugg the code, all the variables have the value 0. Thus, lv_dmbtr, lv_hundred, lv_hundred_check all have the value 0.
May anyone of you know where the problem may be?
Thank you in advance!
Sorry for writing a lot in the code, just wanted to clarify as much as I could what I had done.
yes so I want to display the value of a specific number 700-> seven, 1400-> four.
So the basic formula to get the hundred in a number is the following: Find out how many times 100 fits completely into your number with integer division.
99 / 100 = 0
700 / 100 = 7
701 / 100 = 7
1400 / 100 = 14
1401 / 100 = 14
Now you can simply take this number MOD 10 to get the the individual hundreds.
0 MOD 10 = 0
7 MOD 10 = 7
14 MOD 10 = 4
Keep in mind that ABAP, in contrast to many other programming languages, rounds automatically. So in code this would be:
CONSTANTS lc_hundred TYPE f VALUE '100.0'.
DATA(lv_number) = 1403.
DATA(lv_hundred_count) = CONV i( floor( ( abs( lv_number ) / lc_hundred ) ) MOD 10 ).
I am using SQL Server and I have an int that is 4 to 5 characters long.
I have a report that cast the first 3 digits as the location and last 1 to 2 digits as a cause.
So this is how they look
5142 = 514 = paint line 2 = paint to thin:
50528 = 505 = machining 28 = oblong hole:
SELECT [Suspect]
,left(Suspect,3) as SuspectOP
,Right(Suspect,2) as SuspectID
This query will return
5142 = SuspectOP = 514 SuspectID = 42
50528 = SuspectOP = 505 SuspectID = 28
So what i want is to read everything after the first 3 digits of the int.
Some of the things I have tried are as follows:
Select Cast(Suspect as Varchar(5)),
Substring(Suspect,3,2)
And
Select Suspect % 514 as SuspectID
Which does work as long as the first 3 digits are always 514 which in my case aren't.
You could use a conditional operators based on the length like this:
SELECT
[Suspect]
, SuspectOP = LEFT(Suspect,3)
, SuspectID = CASE
WHEN LEN(Suspect) = 5 THEN RIGHT(Suspect,2)
ELSE RIGHT(Suspect, 1)
END
Mind you, it's not ideal, you should really keep the values separate if your use case is like the one mentioned.
I've been given an example of SQL code and I need to understand what this is doing before I convert it to another language.
Can someone explain to me in English what this code does please?
=IIF(Fields!Time.Value\60 < 10, "0" + CStr(Fields!Time.Value\60), CStr(Fields!Time.Value\60))
+ ":" +
IIF(Fields!Time.Value mod 60 < 10, "0" + FormatNumber(Fields!Time.Value mod 60,0), FormatNumber(Fields!Time.Value mod 60,0))
Many Many Thanks
It is taking a time (in seconds), and converting it to the format "mm:ss". For what it is worth, this looks more like VB script than SQL. Anyway, Cstr simply converts to string, mod gives the remainder of a division, e.g. 16 mod 10 gives you 6, 26 mod 10 would also give you 6.
The first part is using Fields!Time.Value/60 to get the time in minutes, then when this is number is less than 10, appending a 0 to the start:
| If seconds less than 10 | Append 0 to left of seconds | else just use seconds |
=IIF( Fields!Time.Value\60 < 10 , "0" + CStr(Fields!Time.Value\60), CStr(Fields!Time.Value\60))
The next part basically does the same with the seconds, part, but uses mod to get the number of seconds, for example, 97 seconds needs to be broken down to "01:37", so 97 / 60 is used to get the 1, then because this is less than 10, "0" is prepended to it, then 97 mod 60 is used to get the seconds, which gives 37, since this is over 10 nothing is prepended.
I have prices that are going to be converted from one currency to several other currencies but once they are converted I would like to round them to a specific number.
The examples on what I need to round are the following :
Anything under 10 round to the next digit. For this I can just use the CEILING function.
Anything between 10-14 needs to rounded to 14.00. ex: 12.78 to 14.00
Between 14.01 and 15 needs to rounded to 15.00. ex: 14.25 to 15.00
Between 15.01 and 19 needs to be rounded to 19.00 ex: 17.35 to 19.00
Between 19.01 and 20 needs to rounded to 20.00. ex: 19.25 to 20.00
I know this seems a little weird but it is a specification I've been given for my project. To round to the next multiple of 5 I also understand but it is the 4 and 9 values that are really stumping me.
What formula would I need to use to obtain these numbers, or would it be easier to explode the number, grab the value before the decimal and do a case based on the criteria I stated above?
Thanks for your help!
A case seems to make the most sense but your "rounding" logic may look a little strange:
CASE
WHEN value < 10 THEN CEILING(value)
WHEN value <= 14 THEN 14
WHEN value <= 15 THEN 15
WHEN value <= 19 THEN 19
WHEN value <= 20 THEN 20
END
Or you could convert from floats to ints and use the modulus (%) operator:
CASE
WHEN value < 10 THEN CONVERT(int,CEILING(value))
WHEN CONVERT(int,CEILING(value)) % 5 = 0 -- 9.01 - 10, 14.01 - 15, 19.01 - 20, etc.
THEN CONVERT(int,CEILING(value))
WHEN CONVERT(int,CEILING(value)) % 5 <= 4 -- 10.01 - 14, 15.01 - 19, etc.
THEN CONVERT(int,FLOOR(value / 5)) * 5 + 4
END
Since you have slightly odd requirements (there is no simple formula to cover all your cases), then I think your best bet is to use a CASE...WHEN statement like this. Obviously tweak the precise inequalities based on your requirements:
SELECT CASE WHEN colVal < 10 THEN CEILING(colVal)
WHEN colVal <= 14 THEN 14
WHEN colVal <= 15 THEN 15
WHEN colVal <= 19 THEN 19
WHEN colVal <= 20 THEN 20
ELSE someotherval
END
EDIT: Based on the clarified requirements, D Stanley's answer with the modulo stuff is a better fit.