I am trying to find a way of Counting zeros in a rolling using numpy array ?
Using pandas I can get it using:
df['demand'].apply(lambda x: (x == 0).rolling(7).sum()).fillna(0))
or
df['demand'].transform(lambda x: x.rolling(7).apply(lambda x: 7 - np.count _nonzero(x))).fillna(0)
In numpy, using the code from Here
def rolling_window(a, window_size):
shape = (a.shape[0] - window_size + 1, window_size) + a.shape[1:]
print(shape)
strides = (a.strides[0],) + a.strides
return np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides)
arr = np.asarray([10, 20, 30, 5, 6, 0, 0, 0])
np.count_nonzero(rolling_window(arr==0, 7), axis=1)
Output:
array([2, 3])
However, I need the first 6 NaNs as well, and fill it with zeros:
Expected output:
array([0, 0, 0, 0, 0, 0, 2, 3])
Think an efficient one would be with 1D convolution -
def sum_occurences_windowed(arr, W):
K = np.ones(W, dtype=int)
out = np.convolve(arr==0,K)[:len(arr)]
out[:W-1] = 0
return out
Sample run -
In [42]: arr
Out[42]: array([10, 20, 30, 5, 6, 0, 0, 0])
In [43]: sum_occurences_windowed(arr,W=7)
Out[43]: array([0, 0, 0, 0, 0, 0, 2, 3])
Timings on varying length arrays and window of 7
Including count_rolling from #Quang Hoang's post.
Using benchit package (few benchmarking tools packaged together; disclaimer: I am its author) to benchmark proposed solutions.
import benchit
funcs = [sum_occurences_windowed, count_rolling]
in_ = {n:(np.random.randint(0,5,(n)),7) for n in [10,20,50,100,200,500,1000,2000,5000]}
t = benchit.timings(funcs, in_, multivar=True, input_name='Length')
t.plot(logx=True, save='timings.png')
Extending to generic n-dim arrays
from scipy.ndimage.filters import convolve1d
def sum_occurences_windowed_ndim(arr, W, axis=-1):
K = np.ones(W, dtype=int)
out = convolve1d((arr==0).astype(int),K,axis=axis,origin=-(W//2))
out.swapaxes(axis,0)[:W-1] = 0
return out
So, on a 2D array, for counting along each row, use axis=1 and for cols, axis=0 and so on.
Sample run -
In [155]: np.random.seed(0)
In [156]: a = np.random.randint(0,3,(3,10))
In [157]: a
Out[157]:
array([[0, 1, 0, 1, 1, 2, 0, 2, 0, 0],
[0, 2, 1, 2, 2, 0, 1, 1, 1, 1],
[0, 1, 0, 0, 1, 2, 0, 2, 0, 1]])
In [158]: sum_occurences_windowed_ndim(a, W=7)
Out[158]:
array([[0, 0, 0, 0, 0, 0, 3, 2, 3, 3],
[0, 0, 0, 0, 0, 0, 2, 1, 1, 1],
[0, 0, 0, 0, 0, 0, 4, 3, 4, 3]])
# Verify with earlier 1D solution
In [159]: np.vstack([sum_occurences_windowed(i,7) for i in a])
Out[159]:
array([[0, 0, 0, 0, 0, 0, 3, 2, 3, 3],
[0, 0, 0, 0, 0, 0, 2, 1, 1, 1],
[0, 0, 0, 0, 0, 0, 4, 3, 4, 3]])
Let's test out our original 1D input array -
In [187]: arr
Out[187]: array([10, 20, 30, 5, 6, 0, 0, 0])
In [188]: sum_occurences_windowed_ndim(arr, W=7)
Out[188]: array([0, 0, 0, 0, 0, 0, 2, 3])
I would modify the function as follow:
def count_rolling(a, window_size):
shape = (a.shape[0] - window_size + 1, window_size) + a.shape[1:]
strides = (a.strides[0],) + a.strides
rolling = np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides)
out = np.zeros_like(a)
out[window_size-1:] = (rolling == 0).sum(1)
return out
arr = np.asarray([10, 20, 30, 5, 6, 0, 0, 0])
count_rolling(arr,7)
Output:
array([0, 0, 0, 0, 0, 0, 2, 3])
I have a correlation matrix, and I want to get the count of number of items below the diagonal. Preferably in numpy.
[[1, 0, 0, 0, 0],
[.35, 1, 0, 0, 0],
[.42, .31, 1, 0, 0],
[.25, .38, .41, 1, 0],
[.21, .36, .46, .31, 1]]
I want it to return 10. Or, to return the mean of all numbers under the diagonal.
Setup
a = np.array([[1. , 0. , 0. , 0. , 0. ],
[0.35, 1. , 0. , 0. , 0. ],
[0.42, 0.31, 1. , 0. , 0. ],
[0.25, 0.38, 0.41, 1. , 0. ],
[0.21, 0.36, 0.46, 0.31, 1. ]])
numpy.tril_indices will give the indices of all elements under the diagonal (if you provide an offset of -1), and from there, it becomes as simple as indexing and calling mean and size
n, m = a.shape
m = np.tril_indices(n=n, k=-1, m=m)
a[m]
# array([0.35, 0.42, 0.31, 0.25, 0.38, 0.41, 0.21, 0.36, 0.46, 0.31])
a[m].mean()
# 0.346
a[m].size
# 10
A more primitive and bulky answer since numpy provides np.tril_indices as user3483203 mentioned, but what you want per row iteration i is the following (in terms of [row,col] indices):
(i=0)
[1,0] (i=1)
[2,0] [2,1] (i=2)
[3,0] [3,1] [3,2] (i=3)
...
This is essentially the zip of list [i,i,i,...] = [i]*i (i repetitions of i) with [0,1,...,i-1] = range(i). So iterating over the rows of the table, you can actually get the indices per iteration and perform the operator of your choice.
Example setup:
test = np.array(
[[1, 0, 0, 0, 0],
[.35, 1, 0, 0, 0],
[.42, .31, 1, 0, 0],
[.25, .38, .41, 1, 0],
[.21, .36, .46, .31, 1]])
Function definition:
def countdiag(myarray):
numvals = 0
totsum = 0
for i in range(myarray.shape[0]): # row iteration
colc = np.array(range(i)) # calculate column indices
rowc = np.array([i]*i) # calculate row indices
if any(rowc):
print(np.sum(myarray[rowc,colc]))
print(len(myarray[rowc,colc]))
numvals += len(myarray[rowc,colc])
totsum += np.sum(myarray[rowc,colc])
print(list(zip([i]*i, np.arange(i))))
mean = totsum / numvals
return mean, numvals
Test:
[165]: countdiag(test)
[]
0.35
1
[(1, 0)]
0.73
2
[(2, 0), (2, 1)]
1.04
3
[(3, 0), (3, 1), (3, 2)]
1.34
4
[(4, 0), (4, 1), (4, 2), (4, 3)]
0.346
Out[165]:
(0.346, 10)
Well it's not entirely a generic problem, but that's why we're here after all.
A bit of background
My task is to create a tetris like simulation with an AI playing it. The lines do not disappear when they are completed. The end result should be a matrix filled with the neatly placed blocks, with very few or no gaps.
What I chose to do, was a genetic approach, with constant weights and methods for evaluation. The AI would try to place the blocks in all possible places, rotations, evaluate the temporary matrices, and go with the best one.
The problem
In tetris, you can move to the left or right even when the block is on the ground. This allows to solve many positions that would otherwise be impossible. The real problem however, that these holes can even occur mid-air, something like this:
falling J shape, with optimal choice occurring mid-air
The only solution I see here, would be trying all positions, rotations, and all possible combinations of mid-air moves, which I assume is "not an optimal solution" to say it formally.
My question
Is if someone has an idea or another approach, to find these possibilities for placement with realistic amounts of computing power
A single piece can be positioned in a maximum of 10x20x4 = 800 positions on a single board. These will be the nodes of your graph. Where you can move from one node to another in a single move, there is an edge. You can then prune nodes that are illegal (e.g. overlap with existing obstacles on the board). You can also mark nodes as "final" - there, the tile has an obstacle directly under at least one part of it (but they can still have nodes connected to them).
You can then check which final nodes are reachable from the initial node, which is a standard problem.
You could optimize this further by ignoring nodes where the piece is above the current height of the board.
Example code:
import copy
import time
from os import system, name
from random import randint
from wrapt import synchronized
from asciimatics.screen import Screen
import asciimatics
from asciimatics.effects import Cycle, Stars
from asciimatics.renderers import FigletText
from asciimatics.scene import Scene
from asciimatics.screen import Screen
class Tile:
shapes = \
[
[
[0, 0, 2],
[2, 2, 2]
],
[
[3, 3, 0],
[0, 3, 3]
],
[
[0, 4, 4],
[4, 4, 0]
],
[
[5, 0, 0],
[5, 5, 5]
],
[
[6, 6],
[6, 6]
],
[
[0, 0, 0, 0],
[7, 7, 7, 7],
[0, 0, 0, 0]
],
[
[0, 8, 0],
[8, 8, 8]
]
]
def __init__(self, id=-1):
if id >= 0:
self.id = id
else:
self.id = randint(0, len(self.shapes)-1)
self.shape = self.shapes[id]
x = 8
y = 0
id = 0
def rotate(self):
self.shape = list(zip(*self.shape[::-1]))
class Model:
_height = 25
_width = 20
_score = 0
def __init__(self):
self._view = None
self._field = [[0] * self._width for i in range(self._height)]
for i in range(5):
for j in range(self._height):
self._field[j][i] = 1
self._field[j][-i-1] = 1
for i in range(5):
for j in range(self._width):
self._field[-i-1][j] = 1
self._tile = Tile()
self._nexttile = Tile()
def set_view(self, view):
self._view = view
def get_height(self):
i = 0
for r in self._field[:-5]:
full_line = True
if sum(r[5:-5]) > 0:
return i
i += 1
return i
def _merge(self, field, tile):
field_copy = copy.deepcopy(field)
for i in range(len(tile.shape)):
for j in range(len(tile.shape[0])):
field_copy[tile.y + i][tile.x + j] += tile.shape[i][j]
return field_copy
#synchronized
def _is_valid(self, field, tile):
for i in range(len(tile.shape)):
for j in range(len(tile.shape[0])):
if tile.shape[i][j] > 0:
if (field[tile.y + i][tile.x + j] > 0):
return False
return True
def get_board(self):
return self._merge(self._field, self._tile)
def rotate(self):
self._tile.rotate()
if not self._is_valid(self._field, self._tile):
self._tile.rotate()
self._tile.rotate()
self._tile.rotate()
if self._view is not None:
self._view.show()
def _which_lines_completed(self):
lines = []
i = 0
for r in self._field[:-5]:
full_line = True
for c in r:
if c == 0:
full_line = False
if full_line:
lines.append(i)
i += 1
return lines
def _remove_lines(self, lines):
for l in lines:
for i in list(range(1, l+1))[::-1]:
self._field[i] = self._field[i-1].copy()
if len(lines) == 4:
self._score += 5000
elif len(lines) == 3:
self._score += 1000
elif len(lines) == 2:
self._score += 500
elif len(lines) == 1:
self._score += 100
#synchronized
def move_down(self):
self._tile.y += 1
if not self._is_valid(self._field, self._tile):
self._tile.y -= 1
self._field = self._merge(self._field, self._tile)
self._tile = self._nexttile
self._nexttile = Tile()
# Check if any lines need to be removed
lines = self._which_lines_completed()
# If lines need to be removed, notify the view
if len(lines) > 0:
self._view.remove_lines(lines)
# Remove the lines
self._remove_lines(lines)
if self._view is not None:
self._view.show()
#synchronized
def move_left(self):
self._tile.x -= 1
if not self._is_valid(self._field, self._tile):
self._tile.x += 1
else:
if self._view is not None:
self._view.show()
#synchronized
def move_right(self):
self._tile.x += 1
if not self._is_valid(self._field, self._tile):
self._tile.x -= 1
if self._view is not None:
self._view.show()
class AsciimaticView:
def __init__(self, model):
self.screen = Screen.open()
self.model = model
def _show_board(self, board):
#self.screen.clear()
b = board
x = 0
y = 0
for r in b[:-4]:
x = 0
for c in r[4:-4]:
if c == 1:
self.screen.print_at(u'██', x, y, Screen.COLOUR_BLUE, Screen.A_BOLD)
elif c == 2:
self.screen.print_at(u'[]', x, y, Screen.COLOUR_RED, Screen.A_BOLD)
elif c == 3:
self.screen.print_at(u'[]', x, y, Screen.COLOUR_GREEN, Screen.A_BOLD)
elif c == 4:
self.screen.print_at(u'[]', x, y, Screen.COLOUR_GREEN, Screen.A_BOLD)
elif c == 5:
self.screen.print_at(u'[]', x, y, Screen.COLOUR_RED, Screen.A_BOLD)
elif c == 6:
self.screen.print_at(u'[]', x, y, Screen.COLOUR_CYAN, Screen.A_BOLD)
elif c == 7:
self.screen.print_at(u'[]', x, y, Screen.COLOUR_YELLOW, Screen.A_BOLD)
elif c == 8:
self.screen.print_at(u'[]', x, y, Screen.COLOUR_MAGENTA, Screen.A_BOLD)
else:
self.screen.print_at(u' ', x, y, Screen.COLOUR_BLUE, Screen.A_BOLD)
x += 2
y += 1
self.screen.print_at(u' ', 0, y, Screen.COLOUR_RED, Screen.A_BOLD)
self.screen.print_at(u' ', 0, y+1, Screen.COLOUR_RED, Screen.A_BOLD)
self.screen.print_at(u' ', 0, y+2, Screen.COLOUR_RED, Screen.A_BOLD)
self.screen.print_at(u' ', 0, y+3, Screen.COLOUR_RED, Screen.A_BOLD)
for i in range(len(self.model._nexttile.shape)):
x = 0
if (i == 1):
self.screen.print_at(u'Next: ', x, y, Screen.COLOUR_WHITE, Screen.A_BOLD)
x = x + 6
for j in range(len(self.model._nexttile.shape[0])):
c = self.model._nexttile.shape[i][j]
if c == 1:
self.screen.print_at(u'██', x, y, Screen.COLOUR_BLUE, Screen.A_BOLD)
elif c == 2:
self.screen.print_at(u'[]', x, y, Screen.COLOUR_RED, Screen.A_BOLD)
elif c == 3:
self.screen.print_at(u'[]', x, y, Screen.COLOUR_GREEN, Screen.A_BOLD)
elif c == 4:
self.screen.print_at(u'[]', x, y, Screen.COLOUR_GREEN, Screen.A_BOLD)
elif c == 5:
self.screen.print_at(u'[]', x, y, Screen.COLOUR_RED, Screen.A_BOLD)
elif c == 6:
self.screen.print_at(u'[]', x, y, Screen.COLOUR_CYAN, Screen.A_BOLD)
elif c == 7:
self.screen.print_at(u'[]', x, y, Screen.COLOUR_YELLOW, Screen.A_BOLD)
elif c == 8:
self.screen.print_at(u'[]', x, y, Screen.COLOUR_MAGENTA, Screen.A_BOLD)
else:
self.screen.print_at(u' ', x, y, Screen.COLOUR_BLUE, Screen.A_BOLD)
x = x + 2
y = y + 1
x = 0
y = 24
self.screen.print_at(u'Score: ' + str(self.model._score), x, y, Screen.COLOUR_WHITE, Screen.A_BOLD)
self.screen.refresh()
def show(self):
self._show_board(self.model.get_board())
def remove_lines(self, lines):
b = self.model.get_board()
for i in range(5):
for l in lines:
b[l][5:-5] = [1-el for el in b[l][5:-5]]
self._show_board(b)
time.sleep(0.1)
class Node:
x = 0
y = 0
rot = 0
final = False
edges = []
def __eq__(self, other):
"""Overrides the default implementation"""
if isinstance(other, Node):
return (self.x == other.x) and (self.y == other.y) and (self.rot == other.rot)
return False
def is_neighbour(self, other):
if (abs(self.x - other.x) + abs(self.y - other.y) + abs(self.rot - other.rot) == 1) and (other.y >= self.y):
return True
return False
def __hash__(self):
return hash((self.x, self.y, self.rot))
def get_possible_moves(model, tile):
start_node = Node()
start_node.x = model._tile.x
start_node.y = model.get_height() - len(tile.shape)-1
frontier = [start_node]
visited = {start_node: True}
final_nodes = []
while len(frontier) > 0:
n = frontier.pop()
for dx, dy, rot in [(-1, 0, 0), (1, 0, 0), (0, 1, 0), (0, 0, 1)][::-1]:
nn = Node()
nn.x = n.x + dx
nn.y = n.y + dy
nn.rot = (n.rot + rot) % 4
if nn not in visited:
visited[nn] = True
t = Tile(tile.id)
t.x = nn.x
t.y = nn.y
for r in range(nn.rot):
t.rotate()
if model._is_valid(model._field, t):
frontier.append(nn)
# check if node is final
for i in range(len(t.shape)):
for j in range(len(t.shape[0])):
if (t.shape[i][j] > 0) and (model._field[nn.y + i + 1][nn.x + j] > 0):
nn.final = True
final_nodes.append(nn)
break
if (nn.final):
break
print(len(visited))
print(len(final_nodes))
return final_nodes
m = Model()
m._field = [
[1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 3, 3, 0, 3, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 3, 0, 0, 0, 0, 0, 0, 0, 0, 3, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 3, 3, 3, 3, 3, 0, 0, 0, 0, 3, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 3, 3, 3, 3, 3, 0, 0, 0, 3, 3, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
]
m._tile = Tile(3)
import threading
import keyboard
# define a thread which takes input
class InputThread(threading.Thread):
def run(self):
#self.daemon = True
self.last_user_input = None
while True:
try:
if keyboard.is_pressed('left'):#if key 'q' is pressed
self.last_user_input = 'a'
m.move_left()
elif keyboard.is_pressed('right'):#if key 'q' is pressed
self.last_user_input = 'a'
m.move_right()
elif keyboard.is_pressed('z'):
print('z')
self.last_user_input = 'z'
m.rotate()
elif keyboard.is_pressed('down'):
m.move_down()
elif keyboard.is_pressed('q'):
break
else:
pass
# do something based on the user input here
# alternatively, let main do something with
# self.last_user_input
except:
pass
time.sleep(0.1)
# main
#it = InputThread()
#it.start()
fn = get_possible_moves(m, m._tile)
time.sleep(2)
v = AsciimaticView(m)
m.set_view(v)
v.show()
time.sleep(2)
for n in fn:
m._tile = Tile(m._tile.id)
m._tile.x = n.x
m._tile.y = n.y
for r in range(n.rot):
m._tile.rotate()
v.show()
time.sleep(1)
time.sleep(500)