Why am I getting 0 for Series.prod()? - numpy

Question

I have a series consisting of either positive numbers or nan. But when I compute the product, I get 0.
Sample Output :
In [14]: pricerelatives.mean()
Out[14]: 0.99110019490541013
In [15]: pricerelatives.prod()
Out[15]: 0.0
In [16]: len(pricerelatives)
Out[16]: 362698
In [17]: (pricerelatives>0).sum()
Out[17]: 223522
In [18]: (pricerelatives.isnull()).sum()
Out[18]: 139176
In [19]: 223522+139176
Out[19]: 362698
Why I am getting 0 for pricerelatives.prod()?
Update:
Thanks for the quick response. Unfortunately, it did not work:
In [32]: import operator
In [33]: from functools import reduce
In [34]: lst = list(pricerelatives.fillna(1))
In [35]: the_prod = reduce(operator.mul, lst)
In [36]: the_prod
Out[36]: 0.0
Explicitly getting rid of nulls also fails:
In [37]: pricerelatives[pricerelatives.notnull()].prod()
Out[37]: 0.0
Update 2:
Indeed, that's exactly what I just did and was going to add.
In [39]: pricerelatives.describe()
Out[39]:
count 223522.000000
mean 0.991100
std 0.088478
min 0.116398
25% 1.000000
50% 1.000000
75% 1.000000
max 11.062591
dtype: float64
Update 3: Still seems strange to me. So more detailed information:
In [46]: pricerelatives[pricerelatives<1].describe()
Out[46]:
count 50160.000000
mean 0.922993
std 0.083865
min 0.116398
25% 0.894997
50% 0.951488
75% 0.982058
max 1.000000
dtype: float64
Update 4: The ratio is right around your example's cutoff between 0 and >0 but my numbers are much more clustered around 1 than uniform 0,1 and uniform 1,2.
In [52]: 50160./223522
Out[52]: 0.2244074408783028
In [53]: pricerelatives[pricerelatives>=1].describe()
Out[53]:
count 173362.000000
mean 1.010806
std 0.079548
min 1.000000
25% 1.000000
50% 1.000000
75% 1.000000
max 11.062591
dtype: float64
In [54]: pricerelatives[pricerelatives<1].prod()
Out[54]: 0.0

Answer 1

This looks like a "bug" in numpy; see here. It doesn't raise when there's overflow.
Here are some examples:
In [26]: prod(poisson(10, size=30))
Out[26]: -2043494819862020096
In [46]: prod(randn(10000))
Out[46]: 0.0
You'll have to use the long (Python 2) or int (Python 3) type and reduce it using reduce/functools.reduce:
import operator
from functools import reduce
lst = list(pricerelatives.dropna())
the_prod = reduce(operator.mul, lst)
EDIT: It's going to be faster to just remove all of the NaNs and then compute the product rather than setting them to 1 first.
Very informally, the reason you're still getting zero is that the product will approach zero faster as the ratio of the number of values in [0, 1) to values >= 1 grows.
def nnz_ratio(ratio, size=1000):
n1 = ratio * size
n2 = size - n1
s1 = uniform(1, 2, size=n1)
s2 = uniform(0, 1, size=n2)
return Series(hstack((s1, s2)))
ratios = linspace(0.01, 1, 25)
ss = empty(len(ratios))
for i, ratio in enumerate(ratios):
ss[i] = nnz_ratio(ratio).prod()
ss
gives:
array([ 0.0000e+000, 0.0000e+000, 0.0000e+000, 0.0000e+000,
0.0000e+000, 3.6846e-296, 2.6969e-280, 1.2799e-233,
2.0497e-237, 4.9666e-209, 6.5059e-181, 9.8479e-171,
7.7879e-125, 8.2696e-109, 9.3416e-087, 4.1574e-064,
3.9266e-036, 4.1065e+004, 6.6814e+018, 7.1501e+040,
6.2192e+070, 1.3523e+093, 1.0739e+110, 1.5646e+144,
8.6361e+163])
EDIT #2:
If you're computing the geometric mean, use
from scipy.stats import gmean
gm = gmean(pricerelatives.dropna())