I need to transform a date, expressed as a number of seconds since 2000-01-01T00:00:00, to a pandas.Timestamp with a resolution of 1 ns.
I have found two options:
Use: pandas.to_datetime(VALUE, unit='s', epoch=pandas.Timestamp(2000, 1, 1))
Use: epoch=pandas.Timestamp(2000, 1, 1) + pandas.to_timedelta(VALUE, unit='sec')
I was expecting the both of them provide the same result but the results are slightly different, e.g.:
In [2]: Y2K = pandas.Timestamp(2000, 1, 1)
...:
...: s = 538121125.6849735
...:
...: t1 = pandas.to_datetime(s, unit='s', origin=Y2K)
...: t2 = Y2K + pandas.to_timedelta(s, unit='sec')
...:
...: t1 - t2
Out[2]: Timedelta('0 days 00:00:00.000000090')
Am I doing something wrong? Can, this discrepancy, be considered as a bug?
Which is the more correct way to execute this task? Please note that I need a resolution up to 1 ns.
I wouldn't say that's a bug, that's just an incorrect use of pandas.to_datetime method. The second method you proposed seems to be the proper one. It is more accurate because it takes into account the fact that Timestamp is a combination of a date and a time, whereas the first method only takes the date component into account.
Float (double precision) can store only 15 digits (15.9). You use 9 for the integer part, so you can expect only 7 digit precision on decimal part. And you get it as expected.
In any case, do you expect so much precision for any clocks?
As #ChrisQ267 mentioned in the other answer, programs tend to store times in different components, so you have more precision. Common is either date and time in two fields, or second and the decimal part of second fields. So float is not so ideal for high precision timestamps.
But in any case both methods you are using are not precise: both misses leap seconds, so the real result is already off by several seconds (not just the 8 decimal place).
Although both the functions perform the same operation, even they produce same o/p, what is basic difference between these two? Is there any performance related difference, if yes, then which one is better?
Thanks
The documentation is pretty clear on the difference:
NOTE
The REMAINDER function uses the round function in its formula, whereas
the MOD function uses the floor function in its formula.
In other words, when the arguments are positive integers, the mod function returns a positive number between 0 and the second argument. The remainder function returns a number whose absolute value is less than the second argument divided by 2.
The differences can be more striking for negative numbers. One example of a difference is:
REMAINDER(-15, 4)
MOD(-15, 4)
The first gives -3 and the second 1.
EDIT:
What is happening here? How many times does 4 go into -15. One method is "-4" times with a remained of 1. That is: -15 = 4*(-4) + 1. The other is "-3" times: -15 = 4*(-3) - 3.
The difference what is -15/4 expressed as an integer. Using floor, you get -4. Using round, you get -3.
The difference between MOD and REMAINDER function can be understood in the example below:
MOD(13,5): returns 3 whereas, REMAINDER (13,5) returns -2
A simple way to understand the difference is that MOD uses the floor function therefore it counts the occurrence of the second number within the first and returns what is left to complete the first number i.e. 2(5) gives 10 adding 3 gives 13 therefore MOD(13,5)=3
However, the REMAINDER uses a Round function it therefore gets the total number of the second number that could make up the first and then subtract the what makes it excess. i.e. 3(5) = 15 and subtracting 2 gives 13,therefore REMAINDER(13,5)= -2
REMAINDER (-15, 4): 4(-4) gives -16 and adding +1 gives -15 hence REMAINDER (-15,4)=+1
MOD (-15, 4): 3(-4) gives -12, adding -3 gives us -15 hence MOD(15,4)=-3
REMAINDER(-15, 4)--uses round without taking the sign of the number into
consideration.
hence -15/4= -3.75==> round(-3.75)= -4.--(ignore sign during round)
-4*4= -16
-15-(-16)=>1
There fore: REMAINDER(-15, 4)=1
MOD(-15, 4)----uses Floor without taking the sign of the number into
consideration.
-15/4= -3.75==> floor(-3.75)= -3.--(ignore sign in floor)
-3*4=-12
-15-(-12)=>-3
There fore: MOD(-15, 4)= -3
Mod(m,n) is so simple to understand.
Finding Value Mod() output value will always be the manually calculated remainder value when we divide m by n.
Finding Sign The sign of the output remains the same as the first parameter.
eg :
eg 1 : mod (11,3) is the remainder of 11/3 which is 2 and the same sign of 1st parameter. So output is 2
eg 2 : mod (-11,3) is the remainder of 11/3 which is 2 and the same sign of the 1st parameter. So the output is -2
Remainder(m,n) is a bit different
Finding Value You take two multiples of n, such that when multiplied gives the closest lower value and the closest upper value compared to the first parameter(m). The minimum difference between these values will be the output value
Finding Sign The sign of the output value will always be positive if the closest value is less than the first parameter and the if the closest value is greater than the 1st parameter then it will be negative.
eg :
eg 1 : remainder(10,3) The closest multiple values of 3 which are lesser and greater than 10 are 9(3x3) and 12(3x4). And the closest to 10 among 9 & 12 is 9. So the output will be the gap between 9 and 10 which is 1. The closest number is less than the 1st parameter. So the output is 1.
eg 2 : remainder(11,3) The closest multiple values of 11 which are lesser and greater than 11 are 9(3x3) and 12(3x4). And the closest to 12 among 9 & 12 is 12. So the output will be the gap between 12 and 11 which is 1. The closest number is greater than the 1st parameter. So the output is -1.
The question has long had an answer, but I thought some context would be helpful.
https://en.wikipedia.org/wiki/Modulo_operation discusses the “modulo” operation, and the fact that it is based on the remainder after integer division. The problem is that the concept of “remainder” is not clearly defined where there are negative numbers.
Where negative numbers are involved, the difference between modulus and remainder is significant. Mathematically, the modulus operation maps an integer on to a range. This range begins at 0 and is cyclic. For example integer mod 3 is mapped on to the range:
0, 1, 2, 0, 1, 2, …
If the given integer is negative, the cycle is simply extended backwards. If, however, the modulus itself is negative, then the whole range is negative:
0, -2, -1, 0, -2, -1, …
https://rob.conery.io/2018/08/21/mod-and-remainder-are-not-the-same/ illustrates this as a clock with negative numbers.
The practical upshot of this is that the sign of the second number is the same as the sign of the result.
With a remainder, however, you attempt to divide the second number into the first number until you can’t go any more. The remaining difference is called the remainder. If the first number is negative, then, if you stop short, the remainder will be negative.
The practical upshot of which is that the sign of the first number is the sign of the result.
Most coding languages, including most variations of SQL, use a remainder calculation, but very often call it the modulus. And most languages use the notation a % b meaning a mod b.
Oracle, of course, has two functions, which should have been helpful. However, the first function, mod(), actually gives what everybody else calls the remainder. The second, remainder() gives a result which nobody else gives, due to the fact that it is focused on the nearest division. As per the documentation, the mod() functions uses the floor() function, while remainder() uses round().
If you want to compare the results, you can try:
SELECT
remainder(19,5) AS "Remainder ++",
remainder(-19,5) AS "Remainder -+",
remainder(19,-5) AS "Remainder +-",
remainder(-19,-5) AS "Remainder --",
mod(19,5) AS "Mod ++",
mod(-19,5) AS "Mod -+",
mod(19,-5) AS "Mod +-",
mod(-19,-5) AS "Mod --",
mod(mod(19,5) + 5,5) AS "Modulo ++",
mod(mod(-19,5) + 5,5) AS "Modulo ++",
mod(mod(19,-5) + -5,-5) AS "Modulo ++",
mod(mod(-19,-5) + -5,-5) AS "Modulo ++"
FROM DUAL
;
The modulo calculation gives the true modulus, according to https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Remainder
In all, you probably want the remainder as the most natural interpretation, and so you will use mod() to get this result.
When I calculate log(8) / log(2) I get 3 as one would expect:
?log(8)/log(2)
3
However, if I take the int of this calculation like this the result is 2 and thus wrong:
?int(log(8)/log(2))
2
How and why does this happen?
Likely because the actual number returned is of type double. Because floats and doubles cannot accurately represent most base 10 rational numbers the number returned is something like 2.99999999999. Then when you apply int() the .999999999 is truncated.
How floating-point number works: it dedicates a bit for the sign, a few bits to store an exponent, and the rest for the actual fraction. This leads to numbers being represented in a form similar to 1.45 * 10^4; except that instead of the base being 10, it's two.
I have a problem when i'm trying to calculate in a view a formula whose result is smaller than 1.
e.g. I have the next formula: Arenda*TotalArea/10000 as TotalArenda
If I have Arenda=10 and TotalArea=10 I get TotalArenda=0,00 when normally should be 0.01
Thanks
Make Arenda = 10.0 and TotalArea = 10.0 instead of 10 and 10. This will force SQL not to use integer math and you will get your needed accuracy.
In fact, the only way I can get 0.0 as the result is if the Arenda is 10 (integer) while at least one of TotalArea or 10000 contain a decimal and a trailing 0, and only if I override order of operations by grouping using parentheses such as
select 10.0* (10/10000) as blah
If all are integers you get 0. If all contain decimals you get 0.01. If I remove the parentheses, I get 0.01 if ANY of them are non-integer types.
If precision is highly important I would recommend you cast to decimals and not floats:
select CONVERT(decimal(10,2), Arenda) * CONVERT(decimal(10,2), TotalArea) / 10000.0
You are using colunns, so changing the type may not be feasible. SQL Server does integer division on integers (other databases behave differently). Try one of these:
cast(Arenda as float)*cast(TotalArea as float)/10000
or:
Arenda*TotalArea/10000.0
I am doing this small exercise.
declare #No decimal(38,5);
set #No=12345678910111213.14151;
select #No*1000/1000,#No/1000*1000,#No;
Results are:
12345678910111213.141510
12345678910111213.141000
12345678910111213.14151
Why are the results of first 2 selects different when mathematically it should be same?
it is not going to do algebra to convert 1000/1000 to 1. it is going to actually follow the order of operations and do each step.
#No*1000/1000
yields: #No*1000 = 12345678910111213141.51000
then /1000= 12345678910111213.141510
and
#No/1000*1000
yields: #No/1000 = 12345678910111.213141
then *1000= 12345678910111213.141000
by dividing first you lose decimal digits.
because of rounding, the second sql first divides by 1000 which is 12345678910111.21314151, but your decimal is only 38,5, so you lose the last three decimal points.
because when you divide first you get:
12345678910111.21314151
then only six decimal digits are left after point:
12345678910111.213141
then *1000
12345678910111213.141
because the intermediary type is the same as the argument's - in this case decimal(38,5). so dividing first gives you a loss of precision that's reflected in the truncated answer. multiplying by 1000 first doesn't give any loss of precision because that doesn't overload 38 digits.
It's probably because you lose part of data making division first. Notice that #No has 5-point decimal precision so when you divide this number by 1000 you suddenly need 8 digits for decimal part:
123.12345 / 1000 = 0.12312345
So the value has to be rounded (0.12312) and then this value is multiply by 1000 -> 123.12 (you lose 0.00345.
I think that's why the result is what it is...
The first does #No*1000 then divides it by 1000. The intermediates values are always able to represent all the decimal places. The second expression first divides by 1000, which throws away the last two decimal places, before multiplying back to the original value.
You can get around the problem by using CONVERT or CAST on the first value in your expression to increase the number of decimal places and avoid a loss of precision.
DECLARE #num decimal(38,5)
SET #num = 12345678910111213.14151
SELECT CAST(#num AS decimal(38,8)) / 1000 * 1000