While getting GPS data from a receiver I get it in this format 4119.03283,7203.39095. Does anybody know what kind of format that is? I just can't get my mind around what format this is like is it DMS, Decimal?
The format is called DM (Degrees and decimal minutes).
4119.03283,7203.39095
is 41° and 19.03283 minutes (the first two digits in latitude),
same for
7203.39095: 72° 3.39095' (attention here: it should be 3 digits for longitude: so 07203.39095, check if you have 5 digits before comma, then the first 3 are degrees, else the first 2 are degrees. check further for missing leading zerors)
to convert to decimal degrees (format name DEG): 72 + 3.39095 / 60.0
Related
I have a column with datatype float in Teradata. I want to find the Maximum precision and scale for that column.
Note: My column's scale part has more than 10 digits in most of the places.
Sample Data
123.12321323002
13123213.13200003
33232.213123001
The output I need is
Precsion 19 (scale + length of 13123213) and scale is 11 (length of 12321323002)
or
8 (length of 13123213), 11 (length of 12321323002).
I tried to find them buy converting the column as varchar and splitting them based on the '.' and make the integer and fractional part as 2 columns and then finding the max length of 2 columns. But when I'm select the data, Teradata rounds off the scale part. So after that, if I convert them as char, I'm getting lesser value for scale part.
For example:
org data: 1234.12312000123101
data when I select from Teradata: 1234.12312000123
This is a bit long for a comment.
Teradata uses the IEEE format for real/float values. This gives 15-17 digits of precision. Alas, you need 19 digits, so the values you want are not being stored in the database. You cannot recover them.
What you can do is fix the database, so it uses numeric/decimal/number. This supports what you want: NUMERIC(19, 11). You would then need to reload the data so it is correctly stored in the database.
When you need high precision without predefined scale simply switch to the NUMBER datatype, which is a mixture of DECIMAL and FLOAT.
Exact numeric, at least 38 digits precision, no predefined scale, range of 1E-130 .. 1E125.
Float on steroids :-)
I would like to remove zeros after two decimal places in DB2. I have more than 1000 rows for this column
For example
3.6900 needs to be converted to 3.69
I used cast in the query after my research and it gave me the correct result but I would like to understand what is DECIMAL(12,2) and how does this work ? Is there any better way to eliminate zeros?
SELECT CAST(CG.RATE AS DECIMAL(12,2)) AS test from fd.OFFERS CG
Please let me know.
what is DECIMAL(12,2) and how does this work?
The DECIMAL data type represents numbers with a specified decimal precision. You can read a description of the numeric data types:
A DECIMAL number is a packed decimal number with an implicit decimal point. The position of the decimal point is determined by the precision and the scale of the number. The scale, which is the number of digits in the fractional part of the number, cannot be negative or greater than the precision. The maximum precision is 31 digits.
I have a variable of type decimal(26,16) and i want to save a value with 12 digits before decimal point and 14 digits after it, but it raises an "Arithmetic overflow error converting numeric to data type numeric". when i define a variable of this type, what does it really mean? it means it can only store values with 10 digits before and 16 digits after decimal point respectively or something else?
The highest number you can have in a decimal(26,16) is 9999999999.9999999999999999.
So when you try to store a 12 digit number it will to overflow
Try a decimal(28,16) instead
The maximum total number of decimal digits that can be stored, both to the left and to the right of the decimal point. The precision must be a value from 1 through the maximum precision of 38. The default precision is 18.
precision means the maximum number of digit you can use.
scale means the maximum number of digit you can use after decimal(.)
refer this...
http://msdn.microsoft.com/en-us/library/ms187746.aspx
Let's say I have the following 2 queries:
select sum(cast(2666 as float)) * cast(.3 as float)
select sum(cast(2666 as real)) * cast(.3 as real)
The 1st query returns: 799.8
The 2nd query returns: 799.800031781197
Why does the 2nd query not return the same thing as the 1st?
Binary floating point types (like real and float) cannot exactly represent decimal numbers. In particular it is not possible to exactly store 0.3 as a binary floating point number. Instead a number very close to 0.3 is stored. This is called a representation error.
The size of the error is different for real and float because they have different precision.
If you want to store decimal numbers more accurately, consider using decimal or numeric. But note that even though these types can accurately store decimal values up to a certain number of digits, calculations can still produce numbers that cannot be represented exactly. For example the result of 0.1 / 0.3 can not be stored exactly in a decimal even though both 0.1 and 0.3 can. In this case the result will be rounded to the nearest value that can be stored in the type (e.g. 0.333333333 depending on the precision).
i am having hard time determining the length of a Decimal data type. The data i have in column is like 0.08,1.2,12.35,121.36. Now if i go for (2,2) it throws an error : Arithmetic overflow error converting numeric to data type numeric. Just wondering should it be (6,2)? and if yes can anybody tell me Why 6 and 2?
In syntax like
NUMERIC(precision, scale)
precision is the total number of digits (count digits on both sides of the decimal point), and scale is the number of digits to the right of the decimal point.
From your examples, should be NUMERIC(5,2) - meaning five numbers in total and 2 after the decimal point.