I try to measure the execution time of one step within tensorflow estimator while it makes a prediction.These values are then written into a tensor.
That`s my code: (inside the rollout function)
def step_fn(step, time_steps, current_positions, predictions):
"""Simulats one step"""
start_step = time.perf_counter_ns()
n_particles_per_example = tf.convert_to_tensor([num_cargo_particles+num_boundary_particles], dtype = np.int32)
next_position = simulator(current_positions, n_particles_per_example, global_context, particle_types[:,0])
kinematic_mask = get_kinematic_mask(particle_types)
next_position_ground_truth = ground_truth_positions[:, step]
next_position = tf.where(kinematic_mask, next_position, next_position_ground_truth
)
updated_predictions = predictions.write(step, next_position)
next_positions = tf.concat([current_positions[:, 1:],
next_position[:, tf.newaxis]], axis=1)
stop_step = time.perf_counter_ns()
duration = stop_step-start_step
updated_time_steps = time_steps.write(step, duration)
return (step + 1, updated_time_steps, next_positions, updated_predictions)
After execution the tensor looks like that:
[1.7904564e+09 1.7904564e+09 1.7904564e+09 1.7904564e+09 1.7904564e+09
1.7904564e+09 1.7904564e+09 1.7904564e+09 1.7904564e+09 1.7904564e+09
1.7904564e+09 1.7904564e+09 1.7904564e+09 1.7904564e+09 1.7904564e+09
1.7904564e+09 1.7904564e+09 1.7904564e+09 1.7904564e+09 1.7904564e+09
1.7904564e+09 1.7904564e+09 1.7904564e+09 1.7904564e+09 1.7904564e+09
...]
Whats the problem here? Why are all values the same?
I have also tried to use tf.timestamp(),but that did not work either.
(If I write the step number in the tensor, it works, so the tensor is written correctly)
Related
The shape of my data after the mapping function should be (257, 1001, 1). I asserted this condition in the function and the data passed without an issue. But when extracting a vector from the dataset, the shape comes out at (1, 257, 1001, 1). Tfds never fails to be a bloody pain.
The code:
def read_npy_file(data):
# 'data' stores the file name of the numpy binary file storing the features of a particular sound file
# as a bytes string.
# decode() is called on the bytes string to decode it from a bytes string to a regular string
# so that it can passed as a parameter into np.load()
data = np.load(data.decode())
# Shape of data is now (1, rows, columns)
# Needs to be reshaped to (rows, columns, 1):
data = np.reshape(data, (257, 1001, 1))
assert data.shape == (257, 1001, 1), f"Shape of spectrogram is {data.shape}; should be (257, 1001, 1)."
return data.astype(np.float32)
spectrogram_ds = tf.data.Dataset.from_tensor_slices((specgram_files, labels))
spectrogram_ds = spectrogram_ds.map(
lambda file, label: tuple([tf.numpy_function(read_npy_file, [file], [tf.float32]), label]),
num_parallel_calls=tf.data.AUTOTUNE)
num_files = len(train_df)
num_train = int(0.8 * num_files)
num_val = int(0.1 * num_files)
num_test = int(0.1 * num_files)
spectrogram_ds = spectrogram_ds.shuffle(buffer_size=1000)
specgram_train_ds = spectrogram_ds.take(num_train)
specgram_test_ds = spectrogram_ds.skip(num_train)
specgram_val_ds = specgram_test_ds.take(num_val)
specgram_test_ds = specgram_test_ds.skip(num_val)
specgram, _ = next(iter(spectrogram_ds))
# The following assertion raises an error; not the one in the read_npy_file function.
assert specgram.shape == (257, 1001, 1), f"Spectrogram shape is {specgram.shape}. Should be (257, 1001, 1)"
I thought that the first dimension represented the batch size, which is 1, of course, before batching. But after batching by calling batch(batch_size=64) on the dataset, the shape of a batch was (64, 1, 257, 1001, 1) when it should be (64, 257, 1001, 1).
Would appreciate any help.
Although I still can't explain why I'm getting that output, I did find a workaround. I simply reshaped the data in another mapping like so:
def read_npy_file(data):
# 'data' stores the file name of the numpy binary file storing the features of a particular sound file
# as a bytes string.
# decode() is called on the bytes string to decode it from a bytes string to a regular string
# so that it can passed as a parameter into np.load()
data = np.load(data.decode())
# Shape of data is now (1, rows, columns)
# Needs to be reshaped to (rows, columns, 1):
data = np.reshape(data, (257, 1001, 1))
assert data.shape == (257, 1001, 1), f"Shape of spectrogram is {data.shape}; should be (257, 1001, 1)."
return data.astype(np.float32)
specgram_ds = tf.data.Dataset.from_tensor_slices((specgram_files, one_hot_encoded_labels))
specgram_ds = specgram_ds.map(
lambda file, label: tuple([tf.numpy_function(read_npy_file, [file], [tf.float32, ]), label]),
num_parallel_calls=tf.data.AUTOTUNE)
specgram_ds = specgram_ds.map(lambda specgram, label: tuple([tf.reshape(specgram, (257, 1001, 1)), label]),
num_parallel_calls=tf.data.AUTOTUNE)
iterative_process = tff.learning.algorithms.build_unweighted_fed_prox(
model_fn,
proximal_strength= 0.5,
client_optimizer_fn=lambda: tf.keras.optimizers.SGD(learning_rate=0.01),
server_optimizer_fn=lambda: tf.keras.optimizers.SGD(learning_rate=1.0))
state, metrics = iterative_process.next(state, federated_train_data)
print('round 1, metrics={}'.format(metrics))
On executing the round 1, it throws (TypeError: cannot unpack non-iterable LearningProcessOutput object).
It was working fine when we use Fedavg, but not with fedprox
iterative_process.next returns LearningProcessOutput which is not iterable, as the error says.
You can replace it by
output = iterative_process.next(...)
state = output.state
metrics = output.metrics
or just use the output directly.
I fromfile using a structured dtype and have one field that is raw hexbytes ('V2) - it looks like this:
[[b'\x00\x00', b'\x05\x01', b'\x00\x00', b'\x00\x00', b'\x00\x00' .....],
...
[b'\x00\x00', b'\x05\x01', b'\x00\x00', b'\x00\x00', b'\x00\x00' .....]] - the sub-array is shape (44640, 50)
I'd like to decode this entire array into string literals and keep the same shape
(e.g. each 2byte chunk from b'\x05\x01' into '0501')
Tried iterating through using bytes.hex() instance method but doesn't keep the 2bytes x 50 structure ..
Always extremely grateful for your time and advice...
copied from comment with guess as to line breaks
dt3 = np.dtype([('DIG', 'u1', (digField)), ('ANL', 'V2', (anField)), ('MSG', 'u1', (260 - digField - (anField * 2))), ('DAT', 'u1', (20))])
raw_ry = np.fromfile(logpath, dtype=dt3, count=-1)
dt4 = np.dtype('U')
anDecode_ry = np.array([item.hex() for item in raw_ry['ANL']], dtype=dt4)
I have encountered some strange behavior using numpy.asarray with a memoryview object that I can't explain. Here's a brief example using the cython magic in a jupyter notebook -- I simply make a function that expects two struct array buffers and returns them. One has two ints and the other a long and an int:
cdef struct S1:
int iGroup
int iOrder
cdef struct S2:
long iGroup
int iOrder
def test_struct(S1[:] s1, S2[:] s2):
return s1, s2
Now I make two arrays in python to pass to this function:
dt1 = np.dtype([('iGroup', 'i4'), ('iOrder', 'i4')], align=True)
dt2 = np.dtype([('iGroup', 'i8'), ('iOrder', 'i4')], align=True)
a = np.zeros(10, dtype=dt1)
b = np.zeros(10, dtype=dt2)
x, y = test_struct(a,b)
print x
print y
<MemoryView of 'ndarray' object>
<MemoryView of 'ndarray' object>
Both are successfully returned as MemoryView objects. Now I want to turn them into a numpy array:
np.asarray(x)
array([(0, 0), (0, 0), (0, 0), (0, 0), (0, 0), (0, 0), (0, 0), (0, 0),
(0, 0), (0, 0)],
dtype=[('iGroup', '<i4'), ('iOrder', '<i4')])
np.asarray(y)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-204-ca5459515bfd> in <module>()
----> 1 np.asarray(y)
/Users/rok/miniconda/lib/python2.7/site-packages/numpy/core/numeric.pyc in asarray(a, dtype, order)
480
481 """
--> 482 return array(a, dtype, copy=False, order=order)
483
484 def asanyarray(a, dtype=None, order=None):
TypeError: expected a readable buffer object
What am I missing here? Why does this second struct not work? Any tips would be greatly appreciated!
I have written a model serializer class for a model. When I test to see if it serializes properly I get following error.
AssertionError: ReturnDict([(u'id', 1), ('apple', OrderedDict([(u'id', 1), ('data', u'this is a apple data.')])) ...
I tried converting it to dict wrapping aserializer.data to dict function like
dict(aserializer.data)
Then I get following error.
AssertionError: {'status': False, 'http': OrderedDict([(u'id', 1), ...
How to get dictionary data from a serializer.