I have create a table named:
sub
,with field:
addn numeric (24,6) NULL
I have used the values in this txt file to insert.
https://drive.google.com/file/d/1z2ixHDOvHiM5bqDSI3fCbkuXa0Syfjrn/view
Question:
Why is it the if I query this:
select SUM(addn) from sub
Result:
131546008007.610000
and if I paste the result of this in Excel:
select * from sub
the sum is:
131546008007.57
Note:
there are 4 (-0.01) in the query. I don't know if this is trigger and how to solve this
This is topic about Precision, Scale and Length. Numeric datatype is stored/managed in different way than float datatype, for instance.
Try this query, and you will have the same result than in Excel:
select sum(cast(addn as float)) from sub
131546008007.57
Here you have some links where they are explaining that the float datatype is an approximate number, and the decimal is more accurate than the float datatype. So you can see with this than Excel is using approximate numbers.
Precision, Scale, and Length
Here they are explaining than in financial applications you should NOT use floating-point datatypes, so it is good you're using numeric in your DB, and therefore you can rely on your SQL DB in this example.
And here they state that:
Excel was designed in accordance to the IEEE Standard for Binary Floating-Point Arithmetic (IEEE 754). The standard defines how floating-point numbers are stored and calculated. The IEEE 754 standard is widely used because it allows-floating point numbers to be stored in a reasonable amount of space and calculations can occur relatively quickly.
.
Excel store 15 significant digits of precision.
Related
I have a column with datatype float in Teradata. I want to find the Maximum precision and scale for that column.
Note: My column's scale part has more than 10 digits in most of the places.
Sample Data
123.12321323002
13123213.13200003
33232.213123001
The output I need is
Precsion 19 (scale + length of 13123213) and scale is 11 (length of 12321323002)
or
8 (length of 13123213), 11 (length of 12321323002).
I tried to find them buy converting the column as varchar and splitting them based on the '.' and make the integer and fractional part as 2 columns and then finding the max length of 2 columns. But when I'm select the data, Teradata rounds off the scale part. So after that, if I convert them as char, I'm getting lesser value for scale part.
For example:
org data: 1234.12312000123101
data when I select from Teradata: 1234.12312000123
This is a bit long for a comment.
Teradata uses the IEEE format for real/float values. This gives 15-17 digits of precision. Alas, you need 19 digits, so the values you want are not being stored in the database. You cannot recover them.
What you can do is fix the database, so it uses numeric/decimal/number. This supports what you want: NUMERIC(19, 11). You would then need to reload the data so it is correctly stored in the database.
When you need high precision without predefined scale simply switch to the NUMBER datatype, which is a mixture of DECIMAL and FLOAT.
Exact numeric, at least 38 digits precision, no predefined scale, range of 1E-130 .. 1E125.
Float on steroids :-)
I have a problem with round in SQL Server 2014: when I round a number to 2 decimal places sometimes the rounded number is different if I cast to float before or not.
For example, if I execute:
select round(cast(3.945 as float),2)
select round(3.945,2)
I have:
3.94
3.950
But if I execute:
select round(cast(3.935 as float),2)
select round(3.935,2)
I have:
3.94
3.940
It seems incorrect, rounding 3.935 and 3.945 casting to float before, I obtain the same value. Is this a bug?
The problem here is that float is a binary floating point type, where the representation is an approximation of the value. Floats do not losslessly convert to or from base 10, because there is no power of 10 that is also a power of 2. So when this is converted it is done in a way that leaves a roundoff error that pushes the value just before the rounding threshold.
Oddly I cannot reproduce the same behaviour on PostgreSQL and I am not entirely sure why (it may be that on PostgreSQL, round takes a numeric value and this forces a conversion back).
Never use floats where absolute accuracy is required. This occurs not only in databases, but in almost every programming language as well.
As #ChrisTravers says in his answer the issue with rounding a float is that you're not getting exact arithmetic. i.e. That explains why round(3.945,2) rounds up to 3.95 whilst round(3.945E0,2) effectively rounds down to 3.94.
If you're wondering why you see more than 2 decimal places in some cases, that's because of the type you're dealing with. i.e. 3.94 is a float, so doesn't have a specified number of decimal places; whilst 3.950 is the result of rounding a decimal(4,3); which even though we've rounded to 2 decimal places doesn't affect the precision of the type (i.e. it's still decimal(4,3); not converted to decimal(4,2) or decimal(3,2)).
If the purpose of this rounding is for display purposes, you're best of using the str function. i.e.
select str(3.945,4,2) --decimal
select str(3.945E0,4,2) --float
In the above the 4 is the length of the string (i.e. includes the decimal point as a character), and the 2 is the number of decimal places to show.
NB: In this scenario you're chaning the data type to varchar(4).
The below code allows you to see what type you get after performing an operation:
declare #result sql_variant = str(3.945E0,4,2)
select sql_variant_property(#result, 'BaseType') [BaseType]
,sql_variant_property(#result, 'MaxLength') [MaxLength]
,sql_variant_property(#result, 'Precision') [Precision]
,sql_variant_property(#result, 'Scale') [Scale]
I have a table which includes:
COUNT RISK
35 0.6456000000
11 0.5234000000
4 0.8431000000
I need a column to multiply the two columns. However I'm getting the result of:
TOTAL
35
11
4
COUNT - INT
RISK - VARCHAR
SQL is clearly rounding up the decimals as 1. I've tried casting as decimal, numeric and multiplying by 1.0. I need to retain the decimals for an actual calculation. Any help would be great
Convert result to decimal like this
SELECT
CONVERT(DECIMAL(16,10), COUNT * RISK) AS DecimalResult
FROM dbo.whatever;
Or convert COUNT to decimal
SELECT CAST(COUNT AS DECIMAL(16,10)) * RISK
This question is really suspicious. From the surface, it seems the two columns [Count] and [Risk] have different data types with [Count] as integer and [Risk] as decimal or float.
According to BOL, decimal/float data type has higher precedence, I will quote the BOL here
When an operator combines two expressions of different data types, the rules for data type precedence specify that the data type with the lower precedence is converted to the data type with the higher precedence. If the conversion is not a supported implicit conversion, an error is returned. When both operand expressions have the same data type, the result of the operation has that data type
So to me, in SQL Server, when you do
Select [Total]=[Count]*[Risk] from [your_table]
You cannot get the result as shown in the original question.
I'm trying to convert a number to a decimal with two decimals places.
SELECT CONVERT(DECIMAL(10,2),12345)
The above would return 12345.00 but I'm trying to achieve 123.45
You need something like that:
SELECT CONVERT(DECIMAL(15,2),12345/100.0)
SELECT CONVERT(DECIMAL(10,2),CAST(12345 as float)/CAST(100 as float))
Correction: The premise is somewhat flawed, as the data type of a literal number without a decimal point is int, not numeric as implied by the question. In that case, you do need to convert the initial value to either numeric or decimal before dividing:
SELECT CONVERT(DECIMAL,12345)/100
or
SELECT CAST(12345 AS DECIMAL)/100
(cast is the SQL standard, so if you ever want to apply this to other databases, it would be the preferred method.)
Alternately, you can just add a decimal point to the divisor, as SQL server will return the more precise data type when doing arithmetic on heterogeneous types:
SELECT 12345/100.0
According to the documentation, the numeric data type is functionally equivalent to the decimal datatype, so there's really no reason to convert between the two. It seems that all you really want to do is divide the value you have by 100:
SELECT 12345/100
For a SQL int that is being converted to a float, how do I set the precision of the floating point number?
This is the selection I would like to truncate to two or 3 decimal places:
AVG(Cast(e.employee_level as Float))avg_level,
Thanks!
In TSQL, you can specify two different sizes for float, 24 or 53. This will set the precision to 7 or 15 digits respectively.
If all you want to do is truncate to a set number of decimal places, you can use ROUND, ie:
ROUND(AVG(CAST(e.employee_level as float)), 3)
As a general rule, you can't specify the number of digits after the decimal point for a floating-point number. Floating point data types store the closest floating-point approximation to any given value. The closest floating-point approximation is unlikely to have the number of digits you want. Although you might be able to suppress every digit after the third one, that will only change the appearance of the value, not the value itself.
Integers are a different story. An integer--stored, converted, or cast to a floating-point data type--will be stored exactly over a large range. Floating-point data types don't have to store any fractional units for integers.
I'd suggest, though that the best practice for you is to
avoid casting integers to floating-point if you don't need fractional units, or
cast integers to decimal or numeric if you do need fractional units, or
handle display issues entirely in application code.
I have had the same issue when calculating a percentage and needing a resulting string value.
Example: 68 is what % of 379
Result is a float = 17.9419525065900
You can cast/convert to Numeric with the Precision = 2 and get 17.94
If you need the value as a string you can then cast it as a VarChar if needed.
You can use Round() as well but in this case it only makes 17.9419525065900 = 17.9400000000000.
You can also use Ceiling() and Floor() to get the next highest or lowest integer.
Ceiling(17.9419525065900) = 18
Floor(17.9419525065900) = 17
Using these combinations you should be able to achieve a result in any format you need.