I am looking for a julia alternative with the same behavior as more_itertools.consecutive_groups in python.
I came up with a simple implementation but speed is an issue here and I'm not sure if the code is optimized enough.
function consecutive_groups(array)
groups = eltype(array)[]
j = 0
for i=1:length(array)-1
if array[i]+1 != array[i+1]
push!(groups, array[j+1:i])
j = i
end
end
push!(groups, array[j+1:end])
return groups
end
Your implementation is already quite fast. If you know that the consecutive groups will be large you might want to just increase the index instead of pushing every element:
function consecutive_groups_2(v)
n = length(v)
groups = Vector{Vector{eltype(v)}}()
i = j = 1
while i <= n && j <= n
j = i
while j < n && v[j] + 1 == v[j + 1]
j += 1
end
push!(groups,v[i:j])
i = j + 1
end
return groups
end
which is roughly 33% faster on large groups:
julia> x = collect(1:100000);
julia> #btime consecutive_groups(x)
165.939 μs (4 allocations: 781.45 KiB)
1-element Array{Array{Int64,1},1}:
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10 … 99991, 99992, 99993, 99994, 99995, 99996, 99997, 99998, 99999, 100000]
julia> #btime consecutive_groups_2(x)
114.830 μs (4 allocations: 781.45 KiB)
1-element Array{Array{Int64,1},1}:
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10 … 99991, 99992, 99993, 99994, 99995, 99996, 99997, 99998, 99999, 100000]
Related
How to loop through a dataframe series multiple times using a recursive function?
I am trying to get a simple case to work and use it in a more complicated function.
I am using a simple dataframe:
df = pd.DataFrame({'numbers': [1,2,3,4,5]
I want to iterate through the rows multiple time and sum the values. Each iteration, the index starting point increments by 1.
def recursive_sum(df, mysum=0, count=0):
df = df.iloc[count:]
if len(df.index) < 2:
return mysum
else:
for i in range(len(df.index)):
mysum += df.iloc[i, 0]
count += 1
return recursive_sum(df, mysum, count)
I think I should get:
#Iteration 1: count = 0, len(df.index) = 5 < 2, mysum = 1 + 2 + 3 + 4 + 5 = 15
#Iteration 2: count = 1, len(df.index) = 4 < 2, mysum = 15 + 2 + 3 + 4 + 5 = 29
#Iteration 3: count = 2, len(df.index) = 3 < 2, mysum = 29 + 3 + 4 + 5 = 41
#Iteration 4: count = 2, len(df.index) = 2 < 2, mysum = 41 + 4 + 5 = 50
#Iteration 5: count = 2, len(df.index) = 1 < 2, mysum = 50
But I am returning 38.
Just fixed it:
def recursive_sum(df, mysum=0, count=0):
if(len(df.index) - count) < 2:
return mysum
else:
for i in range(count, len(df.index)):
mysum += df.iloc[0]
count += 1
return recursive_sum(df, mysum, count)
I am trying to reduce the memory allocation of an inner loop in my code. Below the part that is not working as expected.
using Random
using StatsBase
using BenchmarkTools
using Distributions
a_dist = Distributions.DiscreteUniform(1, 99)
v_dist = Distributions.DiscreteUniform(1, 2)
population_size = 10000
population = [rand(a_dist, population_size) rand(v_dist, population_size)]
find_all_it3(f::Function, A) = (p[2] for p in eachrow(A) if f(p[1]))
#btime begin
c_pool = find_all_it3(x -> (x < 5), population)
c_pool_dict = countmap(c_pool, alg=:dict)
end
#btime begin
c_pool_indexes = findall(x -> (x < 5) , view(population, :, 1))
c_pool_dict = countmap(population[c_pool_indexes, 2], alg=:dict)
end
I was hoping that the generator (find_all_it3) would not need to allocate much memory.
however as per the btime output it seems that there is an allocation for each loop.
98.040 μs (10006 allocations: 625.64 KiB)
18.894 μs (18 allocations: 11.95 KiB)
Now in my scenario the speed and allocation of the findall eventually become an issue, hence I was trying to find a better alternative through generator/iterators so that less allocation occur; is there a way to do that? Are there options to consider?
I don't have an explaination for it but here are the results of a few tests I made
The best time is achieved with view(population, :, 1) .< 5 (test4)
using broadcast! reduces allocations a bit (test5)
the best way to reduce allocation is to do your own loop (test6)
using BenchmarkTools
using StatsBase
population_size = 10000
population = [rand(1:99, population_size) rand(1:2, population_size)]
find_all_it(f::Function, A) = (p[2] for p in eachrow(A) if f(p[1]))
function test1(population)
c_pool = find_all_it(x -> x < 5, population)
c_pool_dict = countmap(c_pool, alg=:dict)
end
function test3(population)
c_pool_indexes = findall(x -> x < 5, view(population, :, 1))
c_pool_dict = countmap(view(population,c_pool_indexes, 2), alg=:dict)
end
function test4(population)
c_pool_indexes = view(population, :, 1) .< 5
c_pool_dict = countmap(view(population,c_pool_indexes, 2), alg=:dict)
end
function test5(c_pool_indexes, population)
broadcast!(<, c_pool_indexes, view(population, :, 1), 5)
c_pool_dict = countmap(view(population,c_pool_indexes, 2), alg=:dict)
end
function test6(population)
d = Dict{Int,Int}()
for i in eachindex(view(population, :, 1))
if population[i, 1] < 5
d[population[i,2]] = 1 + get(d,population[i,2],0)
end
end
return d
end
julia> #btime test1(population);
68.200 μs (10004 allocations: 625.59 KiB)
julia> #btime test3(population);
14.800 μs (14 allocations: 9.00 KiB)
julia> #btime test4(population);
7.250 μs (8 allocations: 9.33 KiB)
julia> temp = zeros(Bool, population_size);
julia> #btime test5(temp, population);
16.599 μs (5 allocations: 3.78 KiB)
julia> #btime test6(population);
11.299 μs (4 allocations: 608 bytes)
I just started programming in Excel VBA and I have a question.
How can I write a code where the values are 'stepped'?
I want to code this by using do until, do while and for only.
I'm using two variables: x = 1 and y = 1. The value must be the number of the cycle.
The output would look like this:
cells(x,y). value = a
cells(2x,y). value = 2a
cells(2x,2y). value = 3a
cells(3x,2y). value = 4a
cells(3x,3y). value = 5a
cells(4x,3y). value = 6a
cells(4x,4y). value = 7a
This would be an alternative version to do it
Option Explicit
Public Sub Generate()
Dim a As Long: a = 1
Dim i As Long
For i = 1 To 7
Cells((i \ 2) + 1, ((i - 1) \ 2) + 1).Value = i * a
Next i
End Sub
Note that this uses the div Operator 5 \ 2 which is not a standard division 5 / 2. The div operator divides two numbers and returns an integer result.
Standard division of 5 / 2 = 2.5
Div Operator of 5 / 2 = 2
So (i \ 2) + 1, ((i - 1) \ 2) + 1 results in
1, 1
2, 1
2, 2
3, 2
3, 3
4, 3
4, 4
You can step through your sequence in VBA using arrays like so:
Sub StepAXY()
Dim Z As Long, A As Long, X As Long, Y As Long
Dim arrA() As Variant, arrX() As Variant, arrY() As Variant
arrA = Array(1, 2, 3, 4, 5, 6, 7)
arrX = Array(1, 2, 2, 3, 3, 4, 4)
arrY = Array(1, 1, 2, 2, 3, 3, 4)
For Z = 1 To 7 Step 1
A = arrA(Z - 1)
X = arrX(Z - 1)
Y = arrY(Z - 1)
Cells(X * 1, Y * 1) = A * 1
Next Z
End Sub
Here is a different approach. This will give you an extra row (5x,4y) so if this is not desired can be excluded using an If statement.
Sub Generate()
Dim x As Long, y As Long, a As Long, b As Long, c As Long
For b = 1 To 4
Cells(b * x, b * y) = (2 * b - 1) * a
Cells((b + 1) * x, b * y) = 2 * b * a
Next b
End Sub
I have a sorted Series like this:
[2, 4, 5, 6, 8, 9]
I want to produce another Series or ndarray of the same length, where the first two odd numbers and the first two even numbers are labeled sequentially:
[0, 1, 2, _, _, 3]
The _ values I don't really care about. They can be zero.
Now I do it this way:
src = pd.Series([2, 4, 5, 6, 8, 9])
odd = src % 2 != 0
where = np.hstack((np.where(odd)[0][:2], np.where(~odd)[0][:2]))
where.sort() # maintain ordering - thanks to #hpaulj
res = np.zeros(len(src), int)
res[where] = np.arange(len(where))
Can you do it more concisely? The input will never be empty, but there might be no odds or no evens (in which case the result could have length 1, 2, or 3 instead of 4).
Great Problem! I'm still exploring and learning.
I've basically stuck with what you've done so far with modest tweaks for efficiency. I'll update if I think of anything else cool.
conclusions
So far, I've thrashed around alot and haven't improved much.
my answer
fast
odd = src.values % 2
even = 1 - odd
res = ((odd.cumsum() * odd) < 3) * ((even.cumsum() * even) < 3)
(res.cumsum() - 1) * res
alternative 1
pretty fast
a = src.values
odd = (a % 2).astype(bool)
rng = np.arange(len(a))
# same reason these are 2, we have 4 below
where = np.append(rng[~odd][:2], rng[odd][:2])
res = np.zeros(len(a), int)
# nature of the problem necessitates that this is 4
res[where] = np.arange(4)
alternative 2
not as quick, but creative
a = src.values
odd = a % 2
res = np.zeros(len(src), int)
b = np.arange(2)
c = b[:, None] == odd
res[(c.cumsum(1) * c <= 2).all(0)] = np.arange(4)
alternative 3
still slow
odd = src.values % 2
a = (odd[:, None] == [0, 1])
b = ((a.cumsum(0) * a) <= 2).all(1)
(b.cumsum() - 1) * b
timing
code
def pir3(src):
odd = src.values % 2
a = (odd[:, None] == [0, 1])
b = ((a.cumsum(0) * a) <= 2).all(1)
return (b.cumsum() - 1) * b
def pir0(src):
odd = src.values % 2
even = 1 - odd
res = ((odd.cumsum() * odd) < 3) * ((even.cumsum() * even) < 3)
return (res.cumsum() - 1) * res
def pir2(src):
a = src.values
odd = a % 2
res = np.zeros(len(src), int)
c = b[:, None] == odd
res[(c.cumsum(1) * c <= 2).all(0)] = np.arange(4)
return res
def pir1(src):
a = src.values
odd = (a % 2).astype(bool)
rng = np.arange(len(a))
where = np.append(rng[~odd][:2], rng[odd][:2])
res = np.zeros(len(a), int)
res[where] = np.arange(4)
return res
def john0(src):
odd = src % 2 == 0
where = np.hstack((np.where(odd)[0][:2], np.where(~odd)[0][:2]))
res = np.zeros(len(src), int)
res[where] = np.arange(len(where))
return res
def john1(src):
odd = src.values % 2 == 0
where = np.hstack((np.where(odd)[0][:2], np.where(~odd)[0][:2]))
res = np.zeros(len(src), int)
res[where] = np.arange(len(where))
return res
src = pd.Series([2, 4, 5, 6, 8, 9])
src = pd.Series([2, 4, 5, 6, 8, 9] * 10000)
I want to add the values of a vector:
a = np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype='d')
to the values of another vector:
c = np.array([10, 10, 10], dtype='d')
at position given by another array (of the same size of a, with values 0 <= b[i] < len(c))
b = np.array([2, 0, 1, 0, 2, 0, 1, 1, 0, 2], dtype='int32')
This is very simple to write in pseudo code:
for I in range(b.shape[0]):
J = b[I]
c[J] += a[I]
Something like this, but vectorized (length of c is some hundreds in real case).
c[0] += np.sum(a[b==0]) # 27 (10 + 1 + 3 + 5 + 8)
c[1] += np.sum(a[b==1]) # 25 (10 + 2 + 6 + 7)
c[2] += np.sum(a[b==2]) # 23 (10 + 0 + 4 + 9)
My initial guess was:
c[b] += a
but only last values of a are summed.
You can use np.bincount to get ID based weighted summations and then add with c, like so -
np.bincount(b,a) + c