I have an array like this:
arr = np.array([4, 6, 3, 9, 2, 100, 3, 1, 1, 1, 1])
I want to convert it to a bool array like this:
[ T, F, F, F, T, F, T, F, F, T, T]
# 4, 6, 3, 9, 2, 100, 3, 1, 1, 1, 1
I can do it with a loop like this:
mask = np.zeros(len(arr), dtype=bool)
ii = 0
while ii < len(arr):
mask[ii] = True
ii += arr[ii]
It's sort of an indirect indexing scheme, where each element in the input tells us how many subsequent elements are invalid.
How can I do it without using a Python loop, so that it will be fast if the input array is large? I'm happy to use Pandas too.
There may be some vectorization trick I'm not thinking of, but if you can use numba, it's well suited for problems like this - this loop should now be very fast.
import numba
#numba.jit(nopython=True)
def jump_mask(arr):
mask = np.zeros(len(arr), dtype=np.bool_)
ii = 0
while ii < len(arr):
mask[ii] = True
ii += arr[ii]
return mask
Related
I have a torch tensor edge_index of shape (2, N) that represents edges in a graph. For each (x, y) there is also a (y, x), where x and y are node IDs (ints). During the forward pass of my model I need to mask out certain edges. So, for example, I have:
n1 = [0, 3, 4] # list of node ids as x
n2 = [1, 2, 1] # list of node ids as y
edge_index = [[1, 2, 0, 1, 3, 4, 2, 3, 1, 4, 2, 4], # actual edges as (x, y) and (y, x)
[2, 1, 1, 0, 4, 3, 3, 2, 4, 1, 4, 2]]
# do something that efficiently removes (x, y) and (y, x) edges as formed by n1 and n2
Final edge_index should look like:
>>> edge_index
[[1, 2, 3, 4, 2, 4],
[2, 1, 4, 3, 4, 2]]
Preferably we need to efficiently make some kind of boolean mask that I can apply to edge index e.g. as edge_index[:, mask] or something like that.
Could also be done in numpy but I'd like to avoid converting back and forth.
Edit #1:
If that can't be done, then I can think of a way so that, instead of n1 and n2, I have access to the indices of the positions I need to exclude in one tensor e.g. _except=[2, 3, 6, 7, 8, 9] (by making a dict/index once in the beginning).
Is there a way to get the desired result by "telling" edge_index to drop the indices in except? edge_index[:, _except] gives me the ones I want to get rid of. I need its complement operation.
Edit #2:
I managed to do it like this:
mask = torch.ones(edge_index.shape[1], dtype=torch.bool)
for i in range(len(n1)):
mask = mask & ~(torch.tensor([n1[i], n2[i]], dtype=torch.long) == edge_index.T).all(dim=1) & ~(torch.tensor([n2[i], n1[i]], dtype=torch.long) == edge_index.T).all(dim=1)
edge_index[:, mask]
but it is too slow and I can't use it. How can I speed it up?
Edit #3: I managed to solve this Edit#1 efficiently with:
mask = torch.ones(edge_index.shape[1], dtype=torch.bool)
mask[_except] = False
edge_index[:, mask]
Still interested in solving the original problem if someone comes up with something...
If you're ok with the way you suggested at Edit#1,
you get the complement result by:
edge_index[:, [i for i in range(edge_index.shape[1]) if not (i in _except)]]
hope this is fast enough for your requirement.
Edit 1:
from functools import reduce
ids = torch.stack([torch.tensor(n1), torch.tensor(n2)], dim=1)
ids = torch.cat([ids, ids[:, [1,0]]], dim=0)
res = edge_index.unsqueeze(0).repeat(6, 1, 1) == ids.unsqueeze(2).repeat(1, 1, 12)
mask = ~reduce(lambda x, y: x | (reduce(lambda p, q: p & q, y)), res, reduce(lambda p, q: p & q, res[0]))
edge_index[:, mask]
Edit 2:
ids = torch.stack([torch.tensor(n1), torch.tensor(n2)], dim=1)
ids = torch.cat([ids, ids[:, [1,0]]], dim=0)
res = edge_index.unsqueeze(0).repeat(6, 1, 1) == ids.unsqueeze(2).repeat(1, 1, 12)
mask = ~(res.sum(1) // 2).sum(0).bool()
edge_index[:, mask]
I have an array of arrays of shape (n, m), as well as an array b of shape (m). I want to create an array c containing distances to the closest element. I can do it with this code:
a = [[11, 2, 3, 4, 5], [4, 4, 6, 1, -2]]
b = [1, 3, 12, 0, 0]
c = []
for inner in range(len(a[0])):
min_distance = float('inf')
for outer in range(len(a)):
current_distance = abs(b[inner] - a[outer][inner])
if min_distance > current_distance:
min_distance = current_distance
c.append(min_distance)
# c=[3, 1, 6, 1, 2]
Elementwise iteration is very slow. What is the numpy way to do this?
If I understand your goal correctly, I think that this would do:
>>> c = np.min(np.abs(np.array(a) - b), axis = 0)
>>> c
array([3, 1, 6, 1, 2])
I use the following function now, but I don't think it works, but I can't understand the description of the cupy kernel definition. This function is very memory intensive and time-consuming when it comes to huge data.
def cupy_sum(self, bins):
bidx = cupy.cumsum(bins) -1,
return cupy.diff(cupy.r_[0, cupy.cumsum(self)[bidx]])
Refer to other examples and write the following code, do not know if there is a problem.
sum_section_kernel = cp.ElementwiseKernel(
'raw T bins, raw T dats',
'float32 out',
'''
T bin_f = bins[i ];
T bin_l = bins[i+1];
T biv = 0;
for(size_t j=bin_f; j<bin_l; j++){
biv += dats[j];
}
out = biv;
''',
'summe')
a = cp.array([4, 3, 5], dtype=cp.float32)
b = cp.array([1, 1, 1.1, 1, 2, 2, 2, 3, 3, 3, 3, 3], dtype=cp.float32)
y = cp.empty(3, dtype=cp.float32)
a = cp.r_[0,a.cumsum()]
out = sum_section_kernel(a, b, y)
print(out)
> [ 4.100 6.000 15.000]
The example has been put in the above, and the speed has not been improved, but I think there is still the advantage of saving memory.
This question already has answers here:
Is there an efficient way to generate N random integers in a range that have a given sum or average?
(6 answers)
Closed 2 years ago.
I would like to generate a list of 15 integers with sum 12, minimum value is 0 and maximum is 6.
I tried following code
def generate(low,high,total,entity):
while sum(entity)!=total:
entity=np.random.randint(low, high, size=15)
return entity
But above function is not working properly. It is too much time consuming.
Please let me know the efficient way to generate such numbers?
The above will, strictly speaking work. But for 15 numbers between 0 and 6, the odds of generating 12 is not that high. In fact we can calculate the number of possibilities with:
F(s, 1) = 1 for 0≤s≤6
and
F(s, n) = Σ6i=0F(s-i, n-1).
We can calculate that with a value:
from functools import lru_cache
#lru_cache()
def f(s, n, mn, mx):
if n < 1:
return 0
if n == 1:
return int(mn <= s <= mx)
else:
if s < mn:
return 0
return sum(f(s-i, n-1, mn, mx) for i in range(mn, mx+1))
That means that there are 9'483'280 possibilities, out of 4'747'561'509'943 total possibilities to generate a sum of 12, or 0.00019975%. It will thus take approximately 500'624 iterations to come up with such solution.
We thus should better aim to find a straight-forward way to generate such sequence. We can do that by each time calculating the probability of generating a number: the probability of generating i as number as first number in a sequence of n numbers that sums up to s is F(s-i, n-1, 0, 6)/F(s, n, 0, 6). This will guarantee that we generate a uniform list over the list of possibilities, if we would each time draw a uniform number, then it will not match a uniform distribution over the entire list of values that match the given condition:
We can do that recursively with:
from numpy import choice
def sumseq(n, s, mn, mx):
if n > 1:
den = f(s, n, mn, mx)
val, = choice(
range(mn, mx+1),
1,
p=[f(s-i, n-1, mn, mx)/den for i in range(mn, mx+1)]
)
yield val
yield from sumseq(n-1, s-val, mn, mx)
elif n > 0:
yield s
With the above function, we can generate numpy arrays:
>>> np.array(list(sumseq(15, 12, 0, 6)))
array([0, 0, 0, 0, 0, 4, 0, 3, 0, 1, 0, 0, 1, 2, 1])
>>> np.array(list(sumseq(15, 12, 0, 6)))
array([0, 0, 1, 0, 0, 1, 4, 1, 0, 0, 2, 1, 0, 0, 2])
>>> np.array(list(sumseq(15, 12, 0, 6)))
array([0, 1, 0, 0, 2, 0, 3, 1, 3, 0, 1, 0, 0, 0, 1])
>>> np.array(list(sumseq(15, 12, 0, 6)))
array([5, 1, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1])
>>> np.array(list(sumseq(15, 12, 0, 6)))
array([0, 0, 0, 0, 4, 2, 3, 0, 0, 0, 0, 0, 3, 0, 0])
You could try it implementing it a little bit differently.
import random
def generate(low,high,goal_sum,size=15):
output = []
for i in range(size):
new_int = random.randint(low,high)
if sum(output) + new_int <= goal_sum:
output.append(new_int)
else:
output.append(0)
random.shuffle(output)
return output
Also, if you use np.random.randint, your high will actually be high-1
Well, there is a simple and natural solution - use distribution which by definition provides you array of values with the fixed sum. Simplest one is Multinomial Distribution. The only code to add is to check and reject (and repeat sampling) if some sampled value is above maximum.
Along the lines
import numpy as np
def sample_sum_interval(n, p, maxv):
while True:
q = np.random.multinomial(n, p)
v = np.where(q > maxv)
if len(v[0]) == 0: # if len(v) > 0, some values are outside the range, reject
return q
return None
np.random.seed(32345)
k = 15
n = 12
maxv = 6
p = np.full((k), np.float64(1.0)/np.float64(k), dtype=np.float64) # probabilities
q = sample_sum_interval(n, p, maxv)
print(q)
print(np.sum(q))
q = sample_sum_interval(n, p, maxv)
print(q)
print(np.sum(q))
q = sample_sum_interval(n, p, maxv)
print(q)
print(np.sum(q))
UPDATE
I quickly looked at #WillemVanOnsem proposed method, and I believe it is different from multinomial used by myself.
If we look at multinomial PMF, and assume equal probabilities for all k numbers,
p1 = ... = pk = 1/k, then we could write PMF as
PMF(x1,...xk)=n!/(x1!...xk!) p1x1...pkxk =
n!/(x1!...xk!) k-x1...k-xk = n!/(x1!...xk!) k-Sumixi = n!/(x1!...xk!) k-n
Obviously, probabilities of particular x1...xk combinations would be different from each other due to factorials in denominator (modulo permutations, of course), which is different from #WillemVanOnsem approach where all of them would have equal probabilities to appear, I believe.
Moral of the story - those methods produce different distributions.
I have an 1 dimensional sorted array and would like to find all pairs of elements whose difference is no larger than 5.
A naive approach would to be to make N^2 comparisons doing something like
diffs = np.tile(x, (x.size,1) ) - x[:, np.newaxis]
D = np.logical_and(diffs>0, diffs<5)
indicies = np.argwhere(D)
Note here that the output of my example are indices of x. If I wanted the values of x which satisfy the criteria, I could do x[indicies].
This works for smaller arrays, but not arrays of the size with which I work.
An idea I had was to find where there are gaps larger than 5 between consecutive elements. I would split the array into two pieces, and compare all the elements in each piece.
Is this a more efficient way of finding elements which satisfy my criteria? How could I go about writing this?
Here is a small example:
x = np.array([ 9, 12,
21,
36, 39, 44, 46, 47,
58,
64, 65,])
the result should look like
array([[ 0, 1],
[ 3, 4],
[ 5, 6],
[ 5, 7],
[ 6, 7],
[ 9, 10]], dtype=int64)
Here is a solution that iterates over offsets while shrinking the set of candidates until there are none left:
import numpy as np
def f_pp(A, maxgap):
d0 = np.diff(A)
d = d0.copy()
IDX = []
k = 1
idx, = np.where(d <= maxgap)
vidx = idx[d[idx] > 0]
while vidx.size:
IDX.append(vidx[:, None] + (0, k))
if idx[-1] + k + 1 == A.size:
idx = idx[:-1]
d[idx] = d[idx] + d0[idx+k]
k += 1
idx = idx[d[idx] <= maxgap]
vidx = idx[d[idx] > 0]
return np.concatenate(IDX, axis=0)
data = np.cumsum(np.random.exponential(size=10000)).repeat(np.random.randint(1, 20, (10000,)))
pairs = f_pp(data, 1)
#pairs = set(map(tuple, pairs))
from timeit import timeit
kwds = dict(globals=globals(), number=100)
print(data.size, 'points', pairs.shape[0], 'close pairs')
print('pp', timeit("f_pp(data, 1)", **kwds)*10, 'ms')
Sample run:
99963 points 1020651 close pairs
pp 43.00256529124454 ms
Your idea of slicing the array is a very efficient approach. Since your data are sorted you can just calculate the difference and split it:
d=np.diff(x)
ind=np.where(d>5)[0]
pieces=np.split(x,ind)
Here pieces is a list, where you can then use in a loop with your own code on every element.
The best algorithm is highly dependent on the nature of your data which I'm unaware. For example another possibility is to write a nested loop:
pairs=[]
for i in range(x.size):
j=i+1
while x[j]-x[i]<=5 and j<x.size:
pairs.append([i,j])
j+=1
If you want it to be more clever, you can edit the outer loop in a way to jump when j hits a gap.