How to assign more than one box to a job?
Boxes: B1, B2
Jobs: V, W, X, Y, Z
B1 has V, W, and Y running in same sequence.
B2 has W, X and Z running in same sequence.
So, how to put W in B1 and B2 both?
Assigning a single job to multiple box is NOT allowed,
Alternative,
Create two jobs of W such as JOB_W_1 and JOB_W_2 with different names but with the same command and place them in separate boxes
Incase you don't want both the job to run simultaneously, use the condition for JOB_W_1 as notrunning (JOB_W_2) or done (JOB_W_2)
Job Tree would look like
├── BOX_1
│ ├── JOB_V
│ ├── JOB_W_1
│ └── JOB_X
└── BOX_2
├── JOB_W_2
├── JOB_Y
└── JOB_Z
Downstream for any dependency use Success of:
both jobs JOB_W_1 and JOB_W_2 or
both boxes BOX_1 and BOX_2
Related
Spark provides a few different ways to implement UDFs that consume and return Pandas DataFrames. I am currently using the cogrouped version that takes two (co-grouped) Pandas DataFrames as input and returns a third.
For efficient translation between Spark DataFrames and Pandas DataFrames, Spark uses the Apache Arrow memory layout, however transformation is still required to go from Arrow to Pandas and back. I would really like to access the Arrow data directly, as this is how I will ultimately be working with the data in the UDF (using Polars).
It seems wasteful to go from Spark -> Arrow -> Pandas -> Arrow (Polars) on the way in and the reverse on the return.
import pyarrow as pa
import polars as pl
sql_context = SQLContext(spark)
data = [('James',[1, 2]),]
spark_df = sql_context.createDataFrame(data=data, schema = ["name","properties"])
df = pl.from_arrow(pa.Table.from_batches(spark_df._collect_as_arrow()))
print(df)
shape: (1, 2)
┌───────┬────────────┐
│ name ┆ properties │
│ --- ┆ --- │
│ str ┆ list[i64] │
╞═══════╪════════════╡
│ James ┆ [1, 2] │
└───────┴────────────┘
I'm about to do tensorflow serving.
pb file and variable folder are created.
but No file was created under the variable folder.
like this
└── variables
├── variables.data-00000-of-00001
└── variables.index
After further experimentation, I found that the file only occurs when output is output to tf.Variable.
for example
1) z = tf.Variable(3,dtype=tf.float32)
2) z = tf.constant(3,dtype=tf.float32)
1) is created the file but 2) is not created file
z is output variable
signature_def_map= {
"serving_default": tf.saved_model.signature_def_utils.predict_signature_def(
inputs= {"egg": x, "bacon":y},
outputs= {"spam": z})
})
Is it right that I found out?
The above explanation is a test result as a simple example.
This is what I really want to do
sIdSorted = tf.gather(sId, indices[::-1])[0:5]
sess=tf.Session()
print sess.run(sIdSorted,feed_dict={userLat:37.12,userLon:127.2})
As a result of printing, it was output as follows.
['s7' 's1' 's2' 's3' 's4']
However, in this way, nothing is displayed in the variable folder.....
So I tried to output to tf.variable.
sIdSorted = tf.Variable(tf.gather(sId, indices[::-1])[0:5])
but This will output an error to the following.
initial_value must have a shape specified: Tensor("strided_slice_1:0", dtype=string)
so I tried it as follows.
sIdSorted = tf.Variable(tf.constant(tf.gather(sId, indices[::-1])[0:5],shape=[5]))
but This will output an error to the following.
List of Tensors when single Tensor expected
I need your help. Thank you for reading.
**tensorflow version :1.3.0 python 2.x
That is correct: only tf.Variables result in variable files being exported. Those files contain the actual values of the variables. The graph structure itself is stored in the saved_model.pb. That's where your gather (and any other ops) are. You should be able to serve the model.
Is there a way to display the average of multiple different runs on tensorflow?
I can only see them on the same graph (by sending the path of the different runs), but I want to see their average on the graph
As #dga mentioned this is not implemented yet. Here is some code that uses EventAccumulator to combine scalar tensorflow summary values. This can be extended to accommodate the other summary types.
import os
from collections import defaultdict
import numpy as np
import tensorflow as tf
from tensorboard.backend.event_processing.event_accumulator import EventAccumulator
def tabulate_events(dpath):
summary_iterators = [EventAccumulator(os.path.join(dpath, dname)).Reload() for dname in os.listdir(dpath)]
tags = summary_iterators[0].Tags()['scalars']
for it in summary_iterators:
assert it.Tags()['scalars'] == tags
out = defaultdict(list)
for tag in tags:
for events in zip(*[acc.Scalars(tag) for acc in summary_iterators]):
assert len(set(e.step for e in events)) == 1
out[tag].append([e.value for e in events])
return out
def write_combined_events(dpath, d_combined, dname='combined'):
fpath = os.path.join(dpath, dname)
writer = tf.summary.FileWriter(fpath)
tags, values = zip(*d_combined.items())
timestep_mean = np.array(values).mean(axis=-1)
for tag, means in zip(tags, timestep_mean):
for i, mean in enumerate(means):
summary = tf.Summary(value=[tf.Summary.Value(tag=tag, simple_value=mean)])
writer.add_summary(summary, global_step=i)
writer.flush()
dpath = '/path/to/root/directory'
d = tabulate_events(dpath)
write_combined_events(dpath, d)
This solution assumes a directory structure like the following:
dpath
├── 1
│ └── events.out.tfevents.1518552132.Alexs-MacBook-Pro-2.local
├── 11
│ └── events.out.tfevents.1518552180.Alexs-MacBook-Pro-2.local
├── 21
│ └── events.out.tfevents.1518552224.Alexs-MacBook-Pro-2.local
├── 31
│ └── events.out.tfevents.1518552264.Alexs-MacBook-Pro-2.local
└── 41
└── events.out.tfevents.1518552304.Alexs-MacBook-Pro-2.local
Please follow issue 376 to see progress on this. It's an active feature request with some progress in the last month, but as of now, there's not a way to do what you want. Yet.
Since there is still no build in functionality to do this I released a tool for that:
https://github.com/Spenhouet/tensorboard-aggregator
This tool can aggregate multiple tensorboard runs by their max, min, mean, median and standard deviation. The aggregates are either saved in new tensorboard summaries or as .csv files.
I created TensorBoard Reducer to do this with PyTorch.
pip install tensorboard-reducer
tb-reducer runs/of-your-model* -o output-dir -r mean,std,min,max
The aggregation results can be saved to disk either as new TensorBoard logs or CSV / JSON / Excel files.
So say I have an object that has been photographed from multiple angles, how do I tell tensorflow that the images combined is one object and not seperate objects? All tutorials seem to cover individual images per class.
For example this structure:
-vegetables
├───carrot
│ ├───carrot_1
│ │ └───image_1.jpg
│ │ └───image_2.jpg
│ │ └───image_3.jpg
│ └───carrot_2
│ └───image_1.jpg
├───potato
│ ├───potato_1
│ │ └───image_1.jpg
Also in the future I would like to combine these images with more sensor output.
I'm using np.roll() to do nearest-neighbor-like averaging, but I have a feeling there are faster ways. Here is a simplified example, but imagine 3 dimensions and more complex averaging "stencils". Just for example see section 6 of this paper.
Here are a few lines from that simplified example:
for j in range(nper):
phi2 = 0.25*(np.roll(phi, 1, axis=0) +
np.roll(phi, -1, axis=0) +
np.roll(phi, 1, axis=1) +
np.roll(phi, -1, axis=1) )
phi[do_me] = phi2[do_me]
So should I be looking for something that returns views instead of arrays (as it seems roll returns arrays)? In this case is roll initializing a new array each time it's called? I noticed the overhead is huge for small arrays.
In fact it's most efficient for arrays of [100,100] to [300,300] in size on my laptop. Possibly caching issues above that.
Would scipy.ndimage.interpolation.shift() perform better, the way it is implemented here, and if so, is it fixed? In the linked example above, I'm throwing away the wrapped parts anyway, but might not always.
note: in this question I'm looking only for what is available within NumPy / SciPy. Of course there are many good ways to speed up Python and even NumPy, but that's not what I'm looking for here, because I'm really trying to understand NumPy better. Thanks!
np.roll has to create a copy of the array each time, which is why it is (comparatively) slow. Convolution with something like scipy.ndimage.filters.convolve() will be a bit faster, but it may still create copies (depending on the implementation).
In this particular case, we can avoid copying altogether by using numpy views and padding the original array in the beginning.
import numpy as np
def no_copy_roll(nx, ny):
phi_padded = np.zeros((ny+2, nx+2))
# these are views into different sub-arrays of phi_padded
# if two sub-array overlap, they share memory
phi_north = phi_padded[:-2, 1:-1]
phi_east = phi_padded[1:-1, 2:]
phi_south = phi_padded[2:, 1:-1]
phi_west = phi_padded[1:-1, :-2]
phi = phi_padded[1:-1, 1:-1]
do_me = np.zeros_like(phi, dtype='bool')
do_me[1:-1, 1:-1] = True
x0, y0, r0 = 40, 65, 12
x = np.arange(nx, dtype='float')[None, :]
y = np.arange(ny, dtype='float')[:, None]
rsq = (x-x0)**2 + (y-y0)**2
circle = rsq <= r0**2
phi[circle] = 1.0
do_me[circle] = False
n, nper = 100, 100
phi_hold = np.zeros((n+1, ny, nx))
phi_hold[0] = phi
for i in range(n):
for j in range(nper):
phi2 = 0.25*(phi_south +
phi_north +
phi_east +
phi_west)
phi[do_me] = phi2[do_me]
phi_hold[i+1] = phi
return phi_hold
This will cut about 35% off the time for a simple benchmark like
from original import original_roll
from mwe import no_copy_roll
import numpy as np
nx, ny = (301, 301)
arr1 = original_roll(nx, ny)
arr2 = no_copy_roll(nx, ny)
assert np.allclose(arr1, arr2)
here is my profiling result
37.685 <module> timing.py:1
├─ 22.413 original_roll original.py:4
│ ├─ 15.056 [self]
│ └─ 7.357 roll <__array_function__ internals>:2
│ └─ 7.243 roll numpy\core\numeric.py:1110
│ [10 frames hidden] numpy
├─ 14.709 no_copy_roll mwe.py:4
└─ 0.393 allclose <__array_function__ internals>:2
└─ 0.393 allclose numpy\core\numeric.py:2091
[2 frames hidden] numpy
0.391 isclose <__array_function__ internals>:2
└─ 0.387 isclose numpy\core\numeric.py:2167
[4 frames hidden] numpy
For more elaborate stencils this approach still works, but it can get a bit unwieldy. In this case you can take a look at skimage.util.view_as_windows, which uses a variation of this trick (numpy stride tricks) to return an array that gives you cheap access to a window around each element. You will, however, have to do your own padding, and will need to take care to not create copies of the resulting array, which can get expensive fast.
The fastest implementation I could get so far is based on the low-level implementation of scipy.ndimage.interpolation.shift that you already mentioned:
from scipy.ndimage.interpolation import _nd_image, _ni_support
cval = 0.0 # unused for mode `wrap`
mode = _ni_support._extend_mode_to_code('wrap')
_nd_image.zoom_shift(data, None, shift, data, 0, mode, cval) # in-place update
Precomputing mode, cval and shift to call low-level zoom_shift method directly got me about x5 speedup wrt. calling shift instead, and x10 speedup wrt np.roll.