I'm trying to iterate over a dataframe in order to apply a predict function, which calls a Natural Language Model located on GCP. Here is the loop code :
model = 'XXXXXXXXXXXXXXXX'
barometre_df_processed = barometre_df
barometre_df_processed['theme'] = ''
barometre_df_processed['proba'] = ''
print('DEBUT BOUCLE FOR')
for ind in barometre_df.index:
if barometre_df.verbatim[ind] is np.nan :
barometre_df_processed.theme[ind]="RAS"
barometre_df_processed.proba[ind]="1"
else:
print(barometre_df.verbatim[ind])
print(type(barometre_df.verbatim[ind]))
res = get_prediction(file_path={'text_snippet': {'content': barometre_df.verbatim[ind]},'mime_type': 'text/plain'} },model_name=model)
print(res)
theme = res['displayNames']
proba = res["classification"]["score"]
barometre_df_processed.theme[ind]=theme
barometre_df_processed.proba[ind]=proba
and the get_prediction function that I took from the Natural Language AI Documentation :
def get_prediction(file_path, model_name):
options = ClientOptions(api_endpoint='eu-automl.googleapis.com:443')
prediction_client = automl_v1.PredictionServiceClient(client_options=options)
payload = file_path
# Uncomment the following line (and comment the above line) if want to predict on PDFs.
# payload = pdf_payload(file_path)
parameters_dict = {}
params = json_format.ParseDict(parameters_dict, Value())
request = prediction_client.predict(name=model_name, payload=payload, params=params)
print("fonction prediction")
print(request)
return resultat[0]["displayName"], resultat[0]["classification"]["score"], resultat[1]["displayName"], resultat[1]["classification"]["score"], resultat[2]["displayName"], resultat[2]["classification"]["score"]
I'm doing a loop this way because I want each of my couple [displayNames, score] to create a new line on my final dataframe, to have something like this :
verbatim1, theme1, proba1
verbatim1, theme2, proba2
verbatim1, theme3, proba3
verbatim2, theme1, proba1
verbatim2, theme2, proba2
...
The if barometre_df.verbatim[ind] is np.nan is not causing problems, I just use it to deal with nans, don't take care of it.
The error that I have is this one :
TypeError: 'Value' object is not iterable
I guess the issues is about
res = get_prediction(file_path={'text_snippet': {'content': barometre_df.verbatim[ind]} },model_name=model)
but I can't figure what's goign wrong here.
I already try to remove
,'mime_type': 'text/plain'}
from my get_prediction parameters, but it doesn't change anything.
Does someone knows how to deal with this issue ?
Thank you already.
I think you are not iterating correctly.
The way to iterate through a dataframe is:
for index, row in df.iterrows():
print(row['col1'])
I don't know whats going wrong with this small program.
Here is a snippet of the mcve-writer:
def convert_to_example():
example = tf.train.Example(features=tf.train.Features(feature={
'bboxes': _floats_feature([0.,1.])
}))
return example
writer = tf.python_io.TFRecordWriter(output_file)
...
for filename in filenames:
...
example = convert_to_example()
writer.write(example.SerializeToString())
writer.close()
This is how I read the examples:
filename = '/path/to/file'
record_iter = tf.python_io.tf_record_iterator(path=filename)
example = tf.train.Example()
l = []
for record in record_iter:
example.ParseFromString(record)
bboxes = example.features.feature['bboxes'].float_list.value[:]
l.append(bboxes)
print(l)
I have narrowed down the problem in:
it works with bytes_list
it works with int64_list if the list is just one integer but not a list of integers
it works with float_list if the list is just one float but not a list of floats
So, if I use a list of floats/integers, the execution reaches a deadlock or crushes. If I use a float/integer everything runs smooth.
Any idea?
This error is system dependent. On the workstation works just fine but not in my pc. I opened an issue in github.com.
I know that this stackoverflow thread already gives some nice examples about conditionals in tensorflow, but I'm still struggling how to solve my issue of randomly selecting among several different masks in tensorflow.
Right now I can only select between two mask tensors a and b:
rand_num = tf.random_uniform([], minval=0, maxval=2.0, dtype=tf.float32, seed=None)
def if_true():
return b
def if_false():
return a
mask_sel = tf.cond(tf.less(rand_num , tf.constant(1.0)),if_true,if_false)
(I still find it weird that one needs to define these two helper functions, but not using them weirdly throws an error.)
Now the question: Lets say I have 4 mask tensors (a,b,c,d) or more to randomly select, what would be the best way to do that in tensorflow?
In python that would be
rand_num = np.random.uniform(low=0,high=4.0)
if (rand_num < 1.0):
mask_sel = a
elif(rand_num < 2.0):
mask_sel = b
elif(rand_num < 3.0):
mask_sel = c
else
mask_sel = d
About the helper functions, they are useful because they allow tensorflow to know which operations will run under each condition, this way it can optimize by running only the selected branch and ignoring the other. Operations outside the helper functions but used by any of them will always be run before tf.cond runs.
The other options is to use tf.select; you won't need the helper functions here but it will always evaluate both sides before running tf.select which can be inefficient if you don't need to.
Now for the main problem 'selecting from more than 2 tesnors', you can use multiple options:
1- Recursively nesting tf.cond operations:
def select_from_list(selector, tensor_list):
length = len(tensor_list)
if length == 0:
raise ValueError('List is empty')
elif length == 1:
return tensor_list[0]
else:
half = length // 2
return tf.cond(tf.less(selector, float(half)), lambda: select_from_list(selector, tensor_list[:half]), lambda: select_from_list(selector - half, tensor_list[half:]))
2- Using tf.case:
def select_from_list(selector, tensor_list):
length = len(tensor_list)
if length == 0:
raise ValueError('List is empty')
elif length == 1:
return tensor_list[0]
else:
def fn(tensor):
return lambda: tensor
pred_fn_pairs = [(tf.less(selector, float(i+1)), fn(tensor)) for i, tensor in enumerate(tensor_list)]
return tf.case(pred_fn_pairs, default=lambda:tensor_list[-1])
You can test any of them using:
def test(selector, value_list, sess):
return select_from_list(float(selector), [tf.constant(value) for value in value_list]).eval(session = sess)
sess = tf.Session()
test(3.5, [4,2,6,7,5], sess)
This should return 7
Considder the following interactive example
>>> l=imap(str,xrange(1,4))
>>> list(l)
['1', '2', '3']
>>> list(l)
[]
Does anyone know if there is already an implementation somewhere out there with a version of imap (and the other itertools functions) such that the second time list(l) is executed you get the same as the first. And I don't want the regular map because building the entire output in memory can be a waste of memory if you use larger ranges.
I want something that basically does something like
class cmap:
def __init__(self, function, *iterators):
self._function = function
self._iterators = iterators
def __iter__(self):
return itertools.imap(self._function, *self._iterators)
def __len__(self):
return min( map(len, self._iterators) )
But it would be a waste of time to do this manually for all itertools if someone already did this.
ps.
Do you think containers are more zen then iterators since for an iterator something like
for i in iterator:
do something
implicitly empties the iterator while a container you explicitly need to remove elements.
You do not have to build such an object for each type of container. Basically, you have the following:
mkimap = lambda: imap(str,xrange(1,4))
list(mkimap())
list(mkimap())
Now you onlky need a nice wrapping object to prevent the "ugly" function calls. This could work this way:
class MultiIter(object):
def __init__(self, f, *a, **k):
if a or k:
self.create = lambda: f(*a, **k)
else: # optimize
self.create = f
def __iter__(self):
return self.create()
l = MultiIter(lambda: imap(str, xrange(1,4)))
# or
l = MultiIter(imap, str, xrange(1,4))
# or even
#MultiIter
def l():
return imap(str, xrange(1,4))
# and then
print list(l)
print list(l)
(untested, hope it works, but you should get the idea)
For your 2nd question: Iterators and containers both have their uses. You should take whatever best fits your needs.
You may be looking for itertools.tee()
Iterators are my favorite topic ;)
from itertools import imap
class imap2(object):
def __init__(self, f, *args):
self.g = imap(f,*args)
self.lst = []
self.done = False
def __iter__(self):
while True:
try: # try to get something from g
x = next(self.g)
except StopIteration:
if self.done:
# give the old values
for x in self.lst:
yield x
else:
# g was consumed for the first time
self.done = True
return
else:
self.lst.append(x)
yield x
l=imap2(str,xrange(1,4))
print list(l)
print list(l)
So I have this code for an object. That object being a move you can make in a game of rock papers scissor.
Now, the object needs to be both an integer (for matching a protocol) and a string for convenience of writing and viewing.
class Move:
def __init__(self, setMove):
self.numToName = {0:"rock", 1:"paper",2:"scissors"}
self.nameToNum = dict(reversed(pairing) for pairing in self.numToName.items())
if setMove in self.numToName.keys():
self.mMove=setMove
else:
self.mMove=self.nameToNum.get(setMove) #make it to a number
def defeats(self):
return Move((self.mMove-1)%3)
def losesTo(self):
return Move((self.mMove+1)%3)
def tiesWith(self):
return self
#Operator overloading
def __eq__(A,B):
return A.mMove==B.mMove
def __gt__(A,B):
return A.defeats(B)
def __lt__(A,B):
return A.losesTo(B)
def __ge__(A,B):
return A>B or A==B
def __le__(A,B):
return A<B or A==B
def __str__(self):
return self.numToName.get(self.mMove);
def __int__(self):
return self.mMove;
Now I'm kinda new to python, coming from a C and Java background.
A big thing in python is that there is only one correct way to do something.
Another thing is not worrying about type.
I'm pretty explicitly worrying about type here.
So I'm not sure what the correct way to handle these objects is.
At the moment I have an object which an be one of any 3 types (or more but I'm not sure what that would do)
Maybe instead I should be used the objects of different classes? and make them singletons?
Also my object are currently modifiable after creation, which is a bad thing in my mind.
So is this code Pythonic, and how can i make it more elegant?
(I figure this is a good example to use, to help me work out what makes good python code. Sorry if it seems a bit open ended)
To me, the concept of code being "pythonic" really comes down to the idea that once you understand what problem you're trying to solve, the code almost writes itself. In this case, without worrying about the deeper abstractions of players, games, throws, etc., you have the following problem: there are a certain number of types of moves, each with a name, with set rules for which moves beat which other moves, and you need to find a way to define moves and figure out which move wins in a comparison.
When I read your code, I don't immediately see this problem, I see a lot of extra thought that went into the code itself, finding type representations, doing arithmetic tricks, and generally forcing the problem into a code framework, rather than the other way around. So I'd suggest something like:
class Move:
TYPES = ['rock', 'paper', 'scissors']
BEATS = {
'rock': ['scissors'],
'paper': ['rock'],
'scissors': ['paper']
}
def __init__(self, type):
if type not in self.TYPES:
raise Exception("Invalid move type")
self.type = type
def __str__(self):
return self.type
def __cmp__(self, other):
if other.type in self.BEATS[self.type]:
return 1
elif self.type in self.BEATS[other.type]:
return -1
else:
return 0
And you're done. You can throw in all the other accessors, etc. but it's really just icing, the core problem is solved and the code is readable, flexible, easy to extend, etc. That's really what I think "pythonic" means.
Well, you have only three possible moves, right? Why not just represent them as strings? It seems like the only reason you have the numbers at all is to implement the comparisons (i.e. which defeats which) with some "clever" math, but honestly I don't think that's worth it. All you really need is a function to determine which one is the winner in each possible comparison:
def winner(move0, move1):
if move0 == move1:
return None
elif (move0 == 'rock' and move1 == 'scissors') or \
(...paper vs. rock...) or \
(...scissors vs. paper...):
return 0
else:
return 1
I just made up the return values None, 0, and 1 as an example, you could use whatever is appropriate for your situation.
"Simple is better than complex," The Zen of Python line 3 ;-)
Here is a short version that verbalizes the result.
def winner(p1, p2):
actors = ['Paper', 'Scissors', 'Rock']
verbs = {'RoSc':'breaks', 'ScPa':'cut', 'PaRo':'covers'}
p1, p2 = actors.index(p1), actors.index(p2)
winner, looser = ((p1, p2), (p2, p1))[(1,0,1)[p1 - p2]]
return ' '.join([actors[winner],
verbs.get(actors[winner][0:2] + actors[looser][0:2],
'ties'),
actors[looser]])
The benefit of this structure is evident when expanded to cover Rock, Paper, Scissors, Lizard, Spock
def winner(p1, p2):
actors = ['Paper', 'Scissors', 'Spock', 'Lizard', 'Rock']
verbs = {'RoLi':'crushes', 'RoSc':'breaks', 'LiSp':'poisons',
'LiPa':'eats', 'SpSc':'smashes', 'SpRo':'vaporizes',
'ScPa':'cut', 'ScLi':'decapitate', 'PaRo':'covers',
'PaSp':'disproves'}
p1, p2 = actors.index(p1), actors.index(p2)
winner, looser = ((p1, p2), (p2, p1))[(1,0,1,0,1)[p1 - p2]]
return ' '.join([actors[winner],
verbs.get(actors[winner][0:2] + actors[looser][0:2],
'ties'),
actors[looser]])
>>> winner("Rock", "Scissors")
'Rock breaks Scissors'
>>> winner("Rock", "Spock")
'Spock vaporizes Rock'
>>> winner("Spock", "Paper")
'Paper disproves Spock'
>>> winner("Lizard", "Scissors")
'Scissors decapitate Lizard'
>>> winner("Paper", "Paper")
'Paper ties Paper'
mv = {"Scissor":0, "Rock":1, "Paper":2}
def winner(m1, m2):
result = "Tie" if m1 == m2 else max(m1, m2) if abs(m1 - m2) != (len(mv) - 1) else min(m1, m2)
return mv.keys()[mv.values().index(result)] if result in mv.values() else result
I wrote this to prove the concept: with 5 lines and almost no object orientation you can achieve the stated result, paper; rock; scissor.
A dictionary of numbers/strings. If you pass the numbers in, your result will be the name of the winning string. The validity of the win is sequential (a < b < c < a) so you can simply do a distance check to determine the need to override the sequence. I have added "Tie" as that is an obvious case but really constructing the game with players and all is trivial with this method. Now if you want to play Paper, Rock, Scissors, Lizard, Spock we would need to refactor.
I am not sure the game is abstracted well enough. A move is an event which takes two players. In other words, a move is not a player, and player is not a move. What do you think about this:
# notice that the element k+1 defeats element k
THROWS = ['paper', 'scissors', 'rock']
class Player(object):
def __init__(self, name, throws):
# name the player
self.name = name
# the throws are contained a priori
self.throws = throws
def throw(self):
# a throw uses (and removes) the first element of the throws
# list
return self.throw_value(self.throws.pop(0))
def throw_value(self, what):
if what in [0,1,2]:
# if the throw is a legal int, return it
return what
if what in THROWS:
# if the throw is a legal str, return the
# corresponding int
return THROWS.index(what)
# if none of the above, raise error
raise ValueError('invalid throw')
class Game(object):
def __init__(self, player_1, player_2):
# a game has two players
self.player_1 = player_1
self.player_2 = player_2
def go(self, throws=3):
# a "go" of the game throws three times
for _ in range(throws):
print self.throw()
def throw(self):
# a throw contains the rules for winning
value_1 = self.player_1.throw()
value_2 = self.player_2.throw()
if value_1 == value_2:
return 'draw'
if value_1 > value_2:
return self.player_1.name
return self.player_2.name
if __name__ == "__main__":
juan = Player("Juan", ['rock', 0, 'scissors'])
jose = Player("Jose", [1, 'scissors', 2])
game = Game(juan, jose)
game.go()