Related
I search a way to find all the vector from a np.meshgrid(xrange, xrange, xrange) that are related by k = -k.
For the moment I do that :
#numba.njit
def find_pairs(array):
boolean = np.ones(len(array), dtype=np.bool_)
pairs = []
idx = [i for i in range(len(array))]
while len(idx) > 1:
e1 = idx[0]
for e2 in idx:
if (array[e1] == -array[e2]).all():
boolean[e2] = False
pairs.append([e1, e2])
idx.remove(e1)
if e2 != e1:
idx.remove(e2)
break
return boolean, pairs
# Give array of 3D vectors
krange = np.fft.fftfreq(N)
comb_array = np.array(np.meshgrid(krange, krange, krange)).T.reshape(-1, 3)
# Take idx of the pairs k, -k vector and boolean selection that give position of -k vectors
boolean, pairs = find_pairs(array)
It works but the execution time grow rapidly with N...
Maybe someone has already deal with that?
The main problem is that comb_array has a shape of (R, 3) where R = N**3 and the nested loop in find_pairs runs at least in quadratic time since idx.remove runs in linear time and is called in the for loop. Moreover, there are cases where the for loop does not change the size of idx and the loop appear to run forever (eg. with N=4).
One solution to solve this problem in O(R log R) is to sort the array and then check for opposite values in linear time:
import numpy as np
import numba as nb
# Give array of 3D vectors
krange = np.fft.fftfreq(N)
comb_array = np.array(np.meshgrid(krange, krange, krange)).T.reshape(-1, 3)
# Sorting
packed = comb_array.view([('x', 'f8'), ('y', 'f8'), ('z', 'f8')])
idx = np.argsort(packed, axis=0).ravel()
sorted_comb = comb_array[idx]
# Find pairs
#nb.njit
def findPairs(sorted_comb, idx):
n = idx.size
boolean = np.zeros(n, dtype=np.bool_)
pairs = []
cur = n-1
for i in range(n):
while cur >= i:
if np.all(sorted_comb[i] == -sorted_comb[cur]):
boolean[idx[i]] = True
pairs.append([idx[i], idx[cur]])
cur -= 1
break
cur -= 1
return boolean, pairs
findPairs(sorted_comb, idx)
Note that the algorithm assume that for each row, there are only up to one valid matching pair. If there are several equal rows, they are paired 2 by two. If your goal is to extract all the combination of equal rows in this case, then please note that the output will grow exponentially (which is not reasonable IMHO).
This solution is pretty fast even for N = 100. Most of the time is spent in the sort that is not very efficient (unfortunately Numpy does not provide a way to do a lexicographic argsort of the row efficiently yet though this operation is fundamentally expensive).
I try to add the following constraints to my model. my problem: the function g() expects x as a binary numpy array. So the result arr_a depends on the current value of x in every step of the optimization!
Afterwards, I want the max of this array times x to be smaller than 50.
How can I add this constraint dynamically so that arr_a is always rightfully calculated with the value of x at each iteration while telling the model to keep the constraint arr_a * x <= 50 ? Currently I am getting an error when adding the constraint to the model because g() expects x as numpy array to calculate arr_a, arr_b, arr_c ( g uses np.where(x == 1) within its calculation).
#Init model
from ortools.sat.python import cp_model
model = cp_model.CpModel()
# Declare the variables
x = []
for i in range(self.ds.n_banks):
x.append(model.NewIntVar(0, 1, "x[%i]" % (i)))
#add bool vars
a = model.NewBoolVar('a')
arr_a, arr_b, arr_c = g(df1,df2,df3,x)
model.Add((arr_a.astype('int32') * x).max() <= 50).OnlyEnforceIf(a)
model.Add((arr_a.astype('int32') * x).max() > 50).OnlyEnforceIf(a.Not())
Afterwards i add the target function that naturally also depends on x.
model.Minimize(target(x))
def target(x):
arr_a, arr_b, arr_c = g(df1,df2,df3,x)
return (3 * arr_b * x + 2 * arr_c * x).sum()
EDIT:
My problem changed a bit and i managed to get it work without issues. Nevertheless, I experienced that the constraint is never actually met! self-defined-function is a highly non-linear function that expects the indices where x==1 and where x == 0 and returns a numpy array. Also it is not possible to re-build it with pre-defined functions of the sat.solver.
#Init model
model = cp_model.CpModel()
# Declare the variables
x = [model.NewIntVar(0, 1, "x[%i]" % (i)) for i in range(66)]
# add hints
[model.AddHint(x[i],np.random.choice(2, 1, p=[0.4, 0.6])[0]) for i in range(66)]
open_elements = [model.NewBoolVar("open_elements[%i]" % (i)) for i in range(66)]
closed_elements = [model.NewBoolVar("closed_elements[%i]" % (i)) for i in range(6)]
# open indices as bool vars
for i in range(66):
model.Add(x[i] == 1).OnlyEnforceIf(open_elements[i])
model.Add(x[i] != 1).OnlyEnforceIf(open_elements[i].Not())
model.Add(x[i] != 1).OnlyEnforceIf(closed_elements[i])
model.Add(x[i] == 1).OnlyEnforceIf(closed_elements[i].Not())
model.Add((self-defined-function(np.where(open_elements), np.where(closed_elements), some_array).astype('int32') * x - some_vector).all() <= 0)
Even when I apply a simpler function, it will not work properly.
model.Add((self-defined-function(x, some_array).astype('int32') * x - some_vector).all() <= 0)
I also tried the following:
arr_indices_open = []
arr_indices_closed = []
for i in range(66):
if open_elements[i] == True:
arr_indices_open.append(i)
else:
arr_indices_closed.append(i)
# final Constraint
arr_ = self-defined-function(arr_indices_open, arr_indices_closed, some_array)[0].astype('int32')
for i in range(66):
model.Add(arr_[i] * x[i] <= some_other_vector[i])
Some minimal example for the self-defined-function, with which I simply try to say that n_closed shall be smaller than 10. Even that condition is not met by the solver:
def self_defined_function(arr_indices_closed)
return len(arr_indices_closed)
arr_ = self-defined-function(arr_indices_closed)
for i in range(66):
model.Add(arr_ < 10)
I'm not sure I fully understand the question, but generally, if you want to optimize a function g(x), you'll have to implement it in using the solver's primitives (docs).
It's easier to do when your calculation coincides with an existing solver function, e.g.: if you're trying to calculate a linear expression; but could get harder to do when trying to calculate something more complex. However, I believe that's the only way.
I am reading a book, and found an error as below:
def relu(x):
return (x>0)*x
def relu2dev(x):
return (x>0)
street_lights = np.array([[1,0,1],[0,1,1],[0,0,1],[1,1,1]])
walk_stop = np.array([[1,1,0,0]]).T
alpha = 0.2
hidden_size = 4
weights_0_1 = 2*np.random.random((3,hidden_size))-1
weights_1_2 = 2*np.random.random((hidden_size,1))-1
for it in range(60):
layer_2_error = 0;
for i in range(len(street_lights)):
layer_0 = street_lights[i:i+1]
layer_1 = relu(np.dot(layer_0,weights_0_1))
layer_2 = np.dot(layer_1,weights_1_2)
layer_2_delta = (layer_2-walk_stop[i:i+1])
# -> layer_2_delta's shape is (1,1), so why np.sum?
layer_2_error += np.sum((layer_2_delta)**2)
layer_1_delta = layer_2_delta.dot(weights_1_2.T) * relu2dev(layer_1)
weights_1_2 -= alpha * layer_1.T.dot(layer_2_delta)
weights_0_1 -= alpha * layer_0.T.dot(layer_1_delta)
if(it % 10 == 9):
print("Error: " + str(layer_2_error))
The error place is commented with # ->:
layer_2_delta's shape is (1,1), so why would one use np.sum? I think np.sum can be removed, but not quite sure, since it comes from a book.
As you say, layer_2_delta has a shape of (1,1). This means it is a 2 dimensional array with one element: layer_2_delta = np.array([[X]]). However, layer_2_error is a scalar. So you can get the scalar from the array by either selecting the value at the first index (layer_2_delta[0,0]) or by summing all the elements (which in this case is just the one). As the book seems to use "sum of square errors", it seems natural to keep the notation which is square each element in array and then add all of these up (for instruction purposes): this would be more general (e.g., to cases where the layer has more than one element) than the index approach. But you're right, there could be other ways to do this :).
I have a set of points and I want to create line / road network from those points. Firstly, I need to determine the closest point from each of the points. For that, I used the KD Tree and developed a code like this:
def closestPoint(source, X = None, Y = None):
df = pd.DataFrame(source).copy(deep = True) #Ensure source is a dataframe, working on a copy to keep the datasource
if(X is None and Y is None):
raise ValueError ("Please specify coordinate")
elif(not X in df.keys() and not Y in df.keys()):
raise ValueError ("X and/or Y is/are not in column names")
else:
df["coord"] = tuple(zip(df[X],df[Y])) #create a coordinate
if (df["coord"].duplicated):
uniq = df.drop_duplicates("coord")["coord"]
uniqval = list(uniq.get_values())
dupl = df[df["coord"].duplicated()]["coord"]
duplval = list(dupl.get_values())
for kq,vq in uniq.items():
clstu = spatial.KDTree(uniqval).query(vq, k = 3)[1]
df.at[kq,"coord"] = [vq,uniqval[clstu[1]]]
if([uniqval[clstu[1]],vq] in list(df["coord"]) ):
df.at[kq,"coord"] = [vq,uniqval[clstu[2]]]
for kd,vd in dupl.items():
clstd = spatial.KDTree(duplval).query(vd,k = 1)[1]
df.at[kd,"coord"] = [vd,duplval[clstd]]
else:
val = df["coord"].get_values()
for k,v in df["coord"].items():
clst = spatial.KDTree(val).query(vd, k = 3)[1]
df.at[k,"coord"] = [v,val[clst[1]]]
if([val[clst[1]],v] in list (df["coord"])):
df.at[k,"coord"] = [v,val[clst[2]]]
return df["coord"]
The code can return the the closest points around. However, I need to ensure that no double lines are created (e.g (x,y) to (x1,y1) and (x1,y1) to (x,y)) and also I need to ensure that each point can only be used as a starting point of a line and an end point of a line despite the point being the closest one to the other points.
Below is the visualization of the result:
Result of the code
What I want:
What I want
I've also tried to separate the origin and target coordinate and do it like this:
df["coord"] = tuple(zip(df[X],df[Y])) #create a coordinate
df["target"] = "" #create a column for target points
count = 2 # create a count iteration
if (df["coord"].duplicated):
uniq = df.drop_duplicates("coord")["coord"]
uniqval = list(uniq.get_values())
for kq,vq in uniq.items():
clstu = spatial.KDTree(uniqval).query(vq, k = count)[1]
while not vq in (list(df["target"]) and list(df["coord"])):
clstu = spatial.KDTree(uniqval).query(vq, k = count)[1]
df.set_value(kq, "target", uniqval[clstu[count-1]])
else:
count += 1
clstu = spatial.KDTree(uniqval).query(vq, k = count)[1]
df.set_value(kq, "target", uniqval[clstu[count-1]])
but this return an error
IndexError: list index out of range
Can anyone help me with this? Many thanks!
Answering now about the global strategy, here is what I would do (rough pseudo-algorithm):
current_point = one starting point in uniqval
while (uniqval not empty)
construct KDTree from uniqval and use it for next line
next_point = point in uniqval closest to current_point
record next_point as target for current_point
remove current_point from uniqval
current_point = next_point
What you will obtain is a linear graph joining all your points, using closest neighbors "in some way". I don't know if it will fit your needs. You would also obtain a linear graph by taking next_point at random...
It is hard to comment on your global strategy without further detail about the kind of road network your want to obtain. So let me just comment your specific code and explain why the "out of range" error happens. I hope this can help.
First, are you aware that (list_a and list_b) will return list_a if it is empty, else list_b? Second, isn't the condition (vq in list(df["coord"]) always True? If yes, then your while loop is just always executing the else statement, and at the last iteration of the for loop, (count-1) will be greater than the total number of (unique) points. Hence your KDTree query does not return enough points and clstu[count-1] is out of range.
So I have this code for an object. That object being a move you can make in a game of rock papers scissor.
Now, the object needs to be both an integer (for matching a protocol) and a string for convenience of writing and viewing.
class Move:
def __init__(self, setMove):
self.numToName = {0:"rock", 1:"paper",2:"scissors"}
self.nameToNum = dict(reversed(pairing) for pairing in self.numToName.items())
if setMove in self.numToName.keys():
self.mMove=setMove
else:
self.mMove=self.nameToNum.get(setMove) #make it to a number
def defeats(self):
return Move((self.mMove-1)%3)
def losesTo(self):
return Move((self.mMove+1)%3)
def tiesWith(self):
return self
#Operator overloading
def __eq__(A,B):
return A.mMove==B.mMove
def __gt__(A,B):
return A.defeats(B)
def __lt__(A,B):
return A.losesTo(B)
def __ge__(A,B):
return A>B or A==B
def __le__(A,B):
return A<B or A==B
def __str__(self):
return self.numToName.get(self.mMove);
def __int__(self):
return self.mMove;
Now I'm kinda new to python, coming from a C and Java background.
A big thing in python is that there is only one correct way to do something.
Another thing is not worrying about type.
I'm pretty explicitly worrying about type here.
So I'm not sure what the correct way to handle these objects is.
At the moment I have an object which an be one of any 3 types (or more but I'm not sure what that would do)
Maybe instead I should be used the objects of different classes? and make them singletons?
Also my object are currently modifiable after creation, which is a bad thing in my mind.
So is this code Pythonic, and how can i make it more elegant?
(I figure this is a good example to use, to help me work out what makes good python code. Sorry if it seems a bit open ended)
To me, the concept of code being "pythonic" really comes down to the idea that once you understand what problem you're trying to solve, the code almost writes itself. In this case, without worrying about the deeper abstractions of players, games, throws, etc., you have the following problem: there are a certain number of types of moves, each with a name, with set rules for which moves beat which other moves, and you need to find a way to define moves and figure out which move wins in a comparison.
When I read your code, I don't immediately see this problem, I see a lot of extra thought that went into the code itself, finding type representations, doing arithmetic tricks, and generally forcing the problem into a code framework, rather than the other way around. So I'd suggest something like:
class Move:
TYPES = ['rock', 'paper', 'scissors']
BEATS = {
'rock': ['scissors'],
'paper': ['rock'],
'scissors': ['paper']
}
def __init__(self, type):
if type not in self.TYPES:
raise Exception("Invalid move type")
self.type = type
def __str__(self):
return self.type
def __cmp__(self, other):
if other.type in self.BEATS[self.type]:
return 1
elif self.type in self.BEATS[other.type]:
return -1
else:
return 0
And you're done. You can throw in all the other accessors, etc. but it's really just icing, the core problem is solved and the code is readable, flexible, easy to extend, etc. That's really what I think "pythonic" means.
Well, you have only three possible moves, right? Why not just represent them as strings? It seems like the only reason you have the numbers at all is to implement the comparisons (i.e. which defeats which) with some "clever" math, but honestly I don't think that's worth it. All you really need is a function to determine which one is the winner in each possible comparison:
def winner(move0, move1):
if move0 == move1:
return None
elif (move0 == 'rock' and move1 == 'scissors') or \
(...paper vs. rock...) or \
(...scissors vs. paper...):
return 0
else:
return 1
I just made up the return values None, 0, and 1 as an example, you could use whatever is appropriate for your situation.
"Simple is better than complex," The Zen of Python line 3 ;-)
Here is a short version that verbalizes the result.
def winner(p1, p2):
actors = ['Paper', 'Scissors', 'Rock']
verbs = {'RoSc':'breaks', 'ScPa':'cut', 'PaRo':'covers'}
p1, p2 = actors.index(p1), actors.index(p2)
winner, looser = ((p1, p2), (p2, p1))[(1,0,1)[p1 - p2]]
return ' '.join([actors[winner],
verbs.get(actors[winner][0:2] + actors[looser][0:2],
'ties'),
actors[looser]])
The benefit of this structure is evident when expanded to cover Rock, Paper, Scissors, Lizard, Spock
def winner(p1, p2):
actors = ['Paper', 'Scissors', 'Spock', 'Lizard', 'Rock']
verbs = {'RoLi':'crushes', 'RoSc':'breaks', 'LiSp':'poisons',
'LiPa':'eats', 'SpSc':'smashes', 'SpRo':'vaporizes',
'ScPa':'cut', 'ScLi':'decapitate', 'PaRo':'covers',
'PaSp':'disproves'}
p1, p2 = actors.index(p1), actors.index(p2)
winner, looser = ((p1, p2), (p2, p1))[(1,0,1,0,1)[p1 - p2]]
return ' '.join([actors[winner],
verbs.get(actors[winner][0:2] + actors[looser][0:2],
'ties'),
actors[looser]])
>>> winner("Rock", "Scissors")
'Rock breaks Scissors'
>>> winner("Rock", "Spock")
'Spock vaporizes Rock'
>>> winner("Spock", "Paper")
'Paper disproves Spock'
>>> winner("Lizard", "Scissors")
'Scissors decapitate Lizard'
>>> winner("Paper", "Paper")
'Paper ties Paper'
mv = {"Scissor":0, "Rock":1, "Paper":2}
def winner(m1, m2):
result = "Tie" if m1 == m2 else max(m1, m2) if abs(m1 - m2) != (len(mv) - 1) else min(m1, m2)
return mv.keys()[mv.values().index(result)] if result in mv.values() else result
I wrote this to prove the concept: with 5 lines and almost no object orientation you can achieve the stated result, paper; rock; scissor.
A dictionary of numbers/strings. If you pass the numbers in, your result will be the name of the winning string. The validity of the win is sequential (a < b < c < a) so you can simply do a distance check to determine the need to override the sequence. I have added "Tie" as that is an obvious case but really constructing the game with players and all is trivial with this method. Now if you want to play Paper, Rock, Scissors, Lizard, Spock we would need to refactor.
I am not sure the game is abstracted well enough. A move is an event which takes two players. In other words, a move is not a player, and player is not a move. What do you think about this:
# notice that the element k+1 defeats element k
THROWS = ['paper', 'scissors', 'rock']
class Player(object):
def __init__(self, name, throws):
# name the player
self.name = name
# the throws are contained a priori
self.throws = throws
def throw(self):
# a throw uses (and removes) the first element of the throws
# list
return self.throw_value(self.throws.pop(0))
def throw_value(self, what):
if what in [0,1,2]:
# if the throw is a legal int, return it
return what
if what in THROWS:
# if the throw is a legal str, return the
# corresponding int
return THROWS.index(what)
# if none of the above, raise error
raise ValueError('invalid throw')
class Game(object):
def __init__(self, player_1, player_2):
# a game has two players
self.player_1 = player_1
self.player_2 = player_2
def go(self, throws=3):
# a "go" of the game throws three times
for _ in range(throws):
print self.throw()
def throw(self):
# a throw contains the rules for winning
value_1 = self.player_1.throw()
value_2 = self.player_2.throw()
if value_1 == value_2:
return 'draw'
if value_1 > value_2:
return self.player_1.name
return self.player_2.name
if __name__ == "__main__":
juan = Player("Juan", ['rock', 0, 'scissors'])
jose = Player("Jose", [1, 'scissors', 2])
game = Game(juan, jose)
game.go()