If you cast a large enough number with decimal precision to real where the first number in the decimal portion is less than or equal to 1 then the decimal portion is eliminated.
-- decimal precision is captured
-- 3 x 10^5 + 0.1
select 300000.1::real
OUTPUT:
300000.1
-- decimal precision is not captured
-- 3 x 10^6 + 0.1
select 3000000.1::real
OUTPUT:
3000000
-- decimal precision is captured
-- 3 x 10^6 + 0.2
select 3000000.2::real
OUTPUT:
3000000.2
I checked Postgres documentation and see that real dtype is "variable-precision, inexact" with a range of "6 decimal digits precision". I am not sure why decimal precision is captured in the third example but not the second since the size of the number is the same, just a larger decimal portion. Can someone explain this please?
That's just a coincidence.
real is stored in binary, so when you exceed the precision, it's the luck of the draw how far from the intended decimal number the value is.
Related
I'm converting a decimal value with has more than precision which I am converting to
select convert(numeric(8,4),17597.9)
I need to convert this value what is the alternative
The problem is not the precision of the value, but the scale.
The documentation says:
s (scale):
The number of decimal digits that are stored to the right of the decimal point. This number is subtracted from p to determine the maximum number of digits to the left of the decimal point. Scale must be a value from 0 through p, and can only be specified if precision is specified. The default scale is 0 and so 0 <= s <= p. Maximum storage sizes vary, based on the precision.
You've specified a datatype of precision 8, scale 4 - which means that you can have at most 8 digits total, with 4 digits to the right of the decimal point - and by extension of that, a max of 4 digits to the left.
If you want to convert that number, you'll need a to specify a precision p >= s + 5, such as numeric(9,4)
select convert(numeric(9,4),17597.9)
-- 17597.9000
I would like to remove zeros after two decimal places in DB2. I have more than 1000 rows for this column
For example
3.6900 needs to be converted to 3.69
I used cast in the query after my research and it gave me the correct result but I would like to understand what is DECIMAL(12,2) and how does this work ? Is there any better way to eliminate zeros?
SELECT CAST(CG.RATE AS DECIMAL(12,2)) AS test from fd.OFFERS CG
Please let me know.
what is DECIMAL(12,2) and how does this work?
The DECIMAL data type represents numbers with a specified decimal precision. You can read a description of the numeric data types:
A DECIMAL number is a packed decimal number with an implicit decimal point. The position of the decimal point is determined by the precision and the scale of the number. The scale, which is the number of digits in the fractional part of the number, cannot be negative or greater than the precision. The maximum precision is 31 digits.
Let's say I have the following 2 queries:
select sum(cast(2666 as float)) * cast(.3 as float)
select sum(cast(2666 as real)) * cast(.3 as real)
The 1st query returns: 799.8
The 2nd query returns: 799.800031781197
Why does the 2nd query not return the same thing as the 1st?
Binary floating point types (like real and float) cannot exactly represent decimal numbers. In particular it is not possible to exactly store 0.3 as a binary floating point number. Instead a number very close to 0.3 is stored. This is called a representation error.
The size of the error is different for real and float because they have different precision.
If you want to store decimal numbers more accurately, consider using decimal or numeric. But note that even though these types can accurately store decimal values up to a certain number of digits, calculations can still produce numbers that cannot be represented exactly. For example the result of 0.1 / 0.3 can not be stored exactly in a decimal even though both 0.1 and 0.3 can. In this case the result will be rounded to the nearest value that can be stored in the type (e.g. 0.333333333 depending on the precision).
I am facing an error on my SQL script:
Arithmetic overflow error converting numeric to data type numeric
select x.MemberName,x.DOB,x.FilePath,x.Medication,x.NDC,x.Directions,x.Name,x.Strength,x.GenericName,x.QtyOrdered,x.DaysSupply,x.DateFilled,
CASE
WHEN x.test = 0 THEN 'N/A'
WHEN compliance > 100.0 THEN '100.0'
ELSE CONVERT(VARCHAR(5), CAST(FLOOR(compliance *10)/10.0 AS DECIMAL(3,1)))
END as [Compliance]
I am facing the error on just above syntax line.
Here's your problem:
declare #compliance decimal(10,5)
set #compliance = 100.0 -- <----------------
select CAST(FLOOR(#compliance *10)/10.0 AS DECIMAL(3,1))
Throws "Arithmetic overflow error converting numeric to data type numeric" error. Changing to DECIMAL(4,1) works, or as #paola suggests, change your condition to >= 100.0
decimal(p,s):
p (precision) is the maximum total number of decimal digits that will
be stored, both to the left and to the right of the decimal point. The
precision must be a value from 1 through the maximum precision of 38.
The default precision is 18.
s (scale) is the number of decimal digits that will be stored to the
right of the decimal point. This number is subtracted from p to
determine the maximum number of digits to the left of the decimal
point.
In your case decimal(3, 1) means a total of 3 digits with 1 digit to the right of the decimal point,
99.9
whereas decimal(4,1) provides a total of 4 digits with 1 digit to the right of the decimal point,
999.9
This questions has already been answered, but the why is important.
Numeric defines the TOTAL number of digits, and then the number after the decimal.
So DECIMAL(4,1) shows 123.4
DECIMAL(4,3) shows 1.234
In both cases you have a total of 4 digits. In one case you have 1 after the decimal, leaving 3 in front of the decimal. And vice versa.
So I'm dividing an integer by a decimal, and storing the result in a decimal column. However, it always drops the fractional component(the part after the decimal point). If I multiply the result by 10 or 100 I get a more accurate result, but dividing again drops the fractional part again.
The two fields I've inserted into were a precision 5, scale 0 decimal and a precision 5, scale 3 decimal.
I've also tried casting the integer into a decimal and that doesn't make a difference, neither does multiplying by 1.0.
I'm out of ideas or tricks to try.
Thanks, Buzkie
Turns out I was casting incorrectly. After doing using the correct format
CAST(int AS DECIMAL(5,3))
it worked. I had left off the precision and scale before.