Using graph_metrics.py with a saved graph - tensorflow

I want to view statistics of my model by saving my graph to a file then running graph_metrics.py.
I have tried a few different things to write the file, my best effort is:
tf.train.write_graph( session.graph_def, ".", "my_graph", as_text=True )
But here's what happens:
$ python ./util/graph_metrics.py --noinput_binary --graph my_graph
Traceback (most recent call last):
File "./util/graph_metrics.py", line 137, in <module>
tf.app.run()
File ".virtualenv/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv))
File "./util/graph_metrics.py", line 85, in main
FLAGS.batch_size)
File "./util/graph_metrics.py", line 109, in calculate_graph_metrics
input_tensor = sess.graph.get_tensor_by_name(input_layer)
File ".virtualenv/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2531, in get_tensor_by_name
return self.as_graph_element(name, allow_tensor=True, allow_operation=False)
File ".virtualenv/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2385, in as_graph_element
return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
File ".virtualenv/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2427, in _as_graph_element_locked
"graph." % (repr(name), repr(op_name)))
KeyError: "The name 'Mul:0' refers to a Tensor which does not exist. The operation, 'Mul', does not exist in the graph."
Is there a complete working example of saving a graph, then analyzing it with graph_metrics.py?
This process seems to involve a magic incantation that I haven't yet discovered.

The error you're hitting is because you need to specify the name of your own input node with --input_layer= (it just defaults to Mul:0 because that's what we use in one of our Inception models):
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/tools/graph_metrics.py#L51
The graph_metrics script is still very much a work in progress unfortunately, and you may hit problems with shape inference, but hopefully this should get you past the initial hurdle.

Related

running temporal fusion transformer default dataset shape error

I ran default code of Temporal fusion transformer in google colab which downloaded at github.
After clone, when I ran the step 2, there's no way to test training.
python3 -m script_train_fixed_params volatility outputs yes
The problem is shape error in the below.
Computing best validation loss
Computing test loss
/usr/local/lib/python3.7/dist-packages/keras/engine/training_v1.py:2079: UserWarning: `Model.state_updates` will be removed in a future version. This property should not be used in TensorFlow 2.0, as `updates` are applied automatically.
updates=self.state_updates,
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/content/drive/MyDrive/tft_tf2/script_train_fixed_params.py", line 239, in <module>
use_testing_mode=True) # Change to false to use original default params
File "/content/drive/MyDrive/tft_tf2/script_train_fixed_params.py", line 156, in main
targets = data_formatter.format_predictions(output_map["targets"])
File "/content/drive/MyDrive/tft_tf2/data_formatters/volatility.py", line 183, in format_predictions
output[col] = self._target_scaler.inverse_transform(predictions[col])
File "/usr/local/lib/python3.7/dist-packages/sklearn/preprocessing/_data.py", line 1022, in inverse_transform
force_all_finite="allow-nan",
File "/usr/local/lib/python3.7/dist-packages/sklearn/utils/validation.py", line 773, in check_array
"if it contains a single sample.".format(array)
ValueError: Expected 2D array, got 1D array instead:
array=[-1.43120418 1.58885804 0.28558148 ... -1.50945972 -0.16713021
-0.57365613].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.
I've tried to modify code which is predict dataframe shpae of 'data_formatters/volatility.py", line 183, in format_predictions' because I guessed that's where the problem arises.), but I can't handle that.
You have to change line
183 in volatitlity.py
output[col] = self._target_scaler.inverse_transform(predictions[col].values.reshape(-1, 1))
and line 216 in electricity.py
sliced_copy[col] = target_scaler.inverse_transform(sliced_copy[col].values.reshape(-1, 1))
Afterwards the example electricity works fine. And I guess this should be the same with volatility.

Bad Function Call _m5.event.simulate(*args, **kwargs) When Running Full System Benchmarks

It takes nearly three hours for the simulator to get pass the initialization point when I try to run a PARSEC benchmark in full system mode, only to be met with the following output:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "build/X86/python/m5/main.py", line 457, in main
exec(filecode, scope)
File "configs/example/fs.py", line 396, in <module>
Simulation.run(options, root, test_sys, FutureClass)
File "configs/common/Simulation.py", line 726, in run
exit_event = benchCheckpoints(options, maxtick, cptdir)
File "configs/common/Simulation.py", line 269, in benchCheckpoints
exit_event = m5.simulate(maxtick - m5.curTick())
File "build/X86/python/m5/simulate.py", line 180, in simulate
return _m5.event.simulate(*args, **kwargs)
RuntimeError: bad_function_call
I don't even know where to begin solving this problem. Does anyone know what might be causing this issue and how to solve it? Thanks.

Solving MINLP with PYOMO/PYSP

team,
currently I am working on a nonlinear stochastic optimization problem. So far, the toolbox has been really helpful, thank you! However, adding a nonlinear constraint has caused an error. I use the gurobi solver. The problem results from the following constraint.
def max_pcr_power_rule(model, t):
if t == 0:
return 0 <= battery.P_bat_max-model.P_sc_max[t+1]-model.P_pcr
else:
return model.P_trade_c[t+1] + np.sqrt(-2*np.log(rob_opt.max_vio)) \
*sum(model.U_max_pow[t,i]**2 for i in set_sim.tme_dat_stp)**(0.5) \
<= battery.P_bat_max-model.P_sc_max[t+1]-model.P_pcr
model.max_pcr_power = Constraint(set_sim.tme_dat_stp, rule=max_pcr_power_rule)
I receive this error message:
Initializing extensive form algorithm for stochastic programming
problems. Exception encountered. Scenario tree manager attempting to
shut down. Traceback (most recent call last): File
"C:\Users\theil\Anaconda3\Scripts\runef-script.py", line 5, in
sys.exit(pyomo.pysp.ef_writer_script.main()) File "C:\Users\theil\Anaconda3\lib\site-packages\pyomo\pysp\ef_writer_script.py",
line 863, in main
traceback=options.traceback) File "C:\Users\theil\Anaconda3\lib\site-packages\pyomo\pysp\util\misc.py",
line 344, in launch_command
rc = command(options, *cmd_args, **cmd_kwds) File "C:\Users\theil\Anaconda3\lib\site-packages\pyomo\pysp\ef_writer_script.py",
line 748, in runef
ef.solve() File "C:\Users\theil\Anaconda3\lib\site-packages\pyomo\pysp\ef_writer_script.py",
line 430, in solve
**solve_kwds) File "C:\Users\theil\Anaconda3\lib\site-packages\pyomo\opt\parallel\manager.py",
line 122, in queue
return self._perform_queue(ah, *args, **kwds) File "C:\Users\theil\Anaconda3\lib\site-packages\pyomo\opt\parallel\local.py",
line 59, in _perform_queue
results = opt.solve(*args, **kwds) File "C:\Users\theil\Anaconda3\lib\site-packages\pyomo\opt\base\solvers.py",
line 599, in solve
self._presolve(*args, **kwds) File "C:\Users\theil\Anaconda3\lib\site-packages\pyomo\solvers\plugins\solvers\GUROBI.py",
line 224, in _presolve
ILMLicensedSystemCallSolver._presolve(self, *args, **kwds) File "C:\Users\theil\Anaconda3\lib\site-packages\pyomo\opt\solver\shellcmd.py",
line 196, in _presolve
OptSolver._presolve(self, *args, **kwds) File "C:\Users\theil\Anaconda3\lib\site-packages\pyomo\opt\base\solvers.py",
line 696, in _presolve
**kwds) File "C:\Users\theil\Anaconda3\lib\site-packages\pyomo\opt\base\solvers.py",
line 767, in _convert_problem
**kwds) File "C:\Users\theil\Anaconda3\lib\site-packages\pyomo\opt\base\convert.py",
line 110, in convert_problem
problem_files, symbol_map = converter.apply(*tmp, **tmpkw) File "C:\Users\theil\Anaconda3\lib\site-packages\pyomo\solvers\plugins\converter\model.py",
line 96, in apply
io_options=io_options) File "C:\Users\theil\Anaconda3\lib\site-packages\pyomo\core\base\block.py",
line 1681, in write
io_options) File "C:\Users\theil\Anaconda3\lib\site-packages\pyomo\repn\plugins\cpxlp.py",
line 176, in call
include_all_variable_bounds=include_all_variable_bounds) File "C:\Users\theil\Anaconda3\lib\site-packages\pyomo\repn\plugins\cpxlp.py",
line 719, in _print_model_LP
"with nonlinear terms." % (constraint_data.name)) ValueError: Cannot write legal LP file. Constraint '1.max_pcr_power[1]' has a
body with nonlinear terms.
I thought, that the problem may lay within the nested formulation of the constraint, i.e. the combination of sum and exponential terms. Therefore, I put the sum()-term into a separate variable. This didn't change the core the characteristic of the nonlinear constraint, so that the error stayed the same. My other suspicion was, that the problem lays within the gurobi solver. So i tried to utilize ipopt, which produced the follwing error message:
Error evaluating constraint 1: can't evaluate pow'(0,0.5). ERROR:
Solver (ipopt) returned non-zero return code (1) ERROR: See the solver
log above for diagnostic information. Exception encountered. Scenario
tree manager attempting to shut down. Traceback (most recent call
last): File "C:\Users\theil\Anaconda3\Scripts\runef-script.py", line
5, in
sys.exit(pyomo.pysp.ef_writer_script.main()) File "C:\Users\theil\Anaconda3\lib\site-packages\pyomo\pysp\ef_writer_script.py",
line 863, in main
traceback=options.traceback) File "C:\Users\theil\Anaconda3\lib\site-packages\pyomo\pysp\util\misc.py",
line 344, in launch_command
rc = command(options, *cmd_args, **cmd_kwds) File "C:\Users\theil\Anaconda3\lib\site-packages\pyomo\pysp\ef_writer_script.py",
line 748, in runef
ef.solve() File "C:\Users\theil\Anaconda3\lib\site-packages\pyomo\pysp\ef_writer_script.py",
line 434, in solve
**solve_kwds) File "C:\Users\theil\Anaconda3\lib\site-packages\pyomo\opt\parallel\manager.py",
line 122, in queue
return self._perform_queue(ah, *args, **kwds) File "C:\Users\theil\Anaconda3\lib\site-packages\pyomo\opt\parallel\local.py",
line 59, in _perform_queue
results = opt.solve(*args, **kwds) File "C:\Users\theil\Anaconda3\lib\site-packages\pyomo\opt\base\solvers.py",
line 626, in solve
"Solver (%s) did not exit normally" % self.name) pyutilib.common._exceptions.ApplicationError: Solver (ipopt) did not
exit normally
I am wondering now, if my mistake lays within the formulation of the constraint or the way i utilize the solver. Otherwise I have to simplify my problem to make it solvable.
I would be glad, if you can point me in the right direction. Thank you!
Best regards
Philipp
As Erwin mentioned in the comment, Gurobi is generally not intended for nonlinear problems.

Distributed Tensorflow Errors/

When running a distributed tensorflow (TF v0.9.0rc0) set up, I start up 3 parameter servers and then 6 workers. The parameter servers seem to be fine, giving the message Started server with target: grpc://localhost:2222. But the workers give other errors (below) that I have questions about.
It seems to me that sometimes the computers aren't able to communicate with each other, thereby giving the socket error, connection refused errors. It also seems that the workers aren't able to find the parameter servers when initializing their variables and give the Cannot assign a device error.
Can anyone help me out in understanding what theses errors individually mean, how big of a deal each one is, and perhaps give me pointers in how to fix them if needed?
Specifically:
Why am I getting socket errors?
Why are there Master init: Unavailable issues / what do they mean?
How can I ensure that the devices requested are available?
Does this look like something I should post to the issues page of tensorflow's github account?
Notes on setup:
All computers report Tensorflow version: 0.9.0rc0 (python -c "import tensorflow as tf; print(tf.__version__);"),
although a few might have been installed from source instead of the pip packages if that matters.
All computers are on the same 1Gb ethernet switch.
Hardware is mostly the the same, with some workers running dual GPUs.
All of them give this error(ip addreses change):
E0719 12:06:17.711635677 2543 tcp_client_posix.c:173]
failed to connect to 'ipv4:192.168.xx.xx:2222': socket error: connection refused
But all of the non-chief workers also give:
E tensorflow/core/distributed_runtime/master.cc:202] Master init: Unavailable:
Additionally, some of the non-chief workers crash, giving this error:
Traceback (most recent call last):
File "main.py", line 219, in <module>
r.main()
File "main.py", line 119, in main
with sv.prepare_or_wait_for_session(server.target, config=tf.ConfigProto(gpu_options=gpu_options)) as sess:
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py", line 691, in prepare_or_wait_for_sessionn max_wait_secs=max_wait_secs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/session_manager.py", line 282, in wait_for_session
sess.run([self._local_init_op])
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 372, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 636, in _run
feed_dict_string, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 708, in _do_run
target_list, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 728, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.InvalidArgumentError: Cannot assign a device to node 'save/restore_slice_23':
Could not satisfy explicit device specification '/job:ps/task:3/device:CPU:0'
because no devices matching that specification are registered in this process; available devices:
/job:ps/replica:0/task:0/cpu:0,
/job:ps/replica:0/task:1/cpu:0,
/job:ps/replica:0/task:2/cpu:0,
/job:ps/replica:0/task:4/cpu:0,
/job:worker/replica:0/task:0/cpu:0,
/job:worker/replica:0/task:0/gpu:0,
/job:worker/replica:0/task:1/cpu:0,
/job:worker/replica:0/task:1/gpu:0,
/job:worker/replica:0/task:2/cpu:0,
/job:worker/replica:0/task:2/gpu:0
[[Node: save/restore_slice_23 = RestoreSlice[dt=DT_FLOAT, preferred_shard=-1, _device="/job:ps/task:3/device:CPU:0"](save/Const, save/restore_slice_23/tensor_name, save/restore_slice_23/shape_and_slice)]]
Caused by op u'save/restore_slice_23', defined at:
File "main.py", line 219, in <module>
r.main()
File "main.py", line 101, in main
saver = tf.train.Saver()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 845, in __init__
restore_sequentially=restore_sequentially)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 515, in build
filename_tensor, vars_to_save, restore_sequentially, reshape)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 271, in _AddRestoreOps
values = self.restore_op(filename_tensor, vs, preferred_shard)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 186, in restore_op
preferred_shard=preferred_shard)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/io_ops.py", line 202, in _restore_slice
preferred_shard, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_io_ops.py", line 358, in _restore_slice
preferred_shard=preferred_shard, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/op_def_library.py", line 704, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2260, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1230, in __init__
self._traceback = _extract_stack()
I figured out what my problem was.
TL;DR: The chief needs to know about all the variables in order to initialize them all. Non-chief workers can't create their own variables.
I was converting an old program where all workers had a few independent variables, but needed to share some variables (I was using ZMQ to pass these) to a distributed TensorFlow setup, and forgot to initialize all of the variables on all of the workers. I had something like
# Create worker specific variable
with tf.variable_scope("world_{}".format(**worker_id**)):
w1 = tf.get_variable("weight", shape=(input_dim, hidden_dim), dtype=tf.float32, initializer=tf.truncated_normal_initializer())
instead of doing something like this:
# Create all worker specific variables
all_w1 = {}
for worker in worker_cnt:
with tf.variable_scope("world_{}".format(**worker_id**)):
all_w1[worker] = tf.get_variable("weight", shape=(input_dim, hidden_dim), dtype=tf.float32, initializer=tf.truncated_normal_initializer())
# grab worker specific variable
w1 = all_w1[**worker_id**]
As for the errors...
I suspect that this caused some workers to die with the Master init: Unavailable: error message above because the chief never knew about the variables the workers wanted to create.
I don't have a solid explanation for why the devices unavailable (3rd) error didn't find that device, but I think it's again, because only the master could create that, and he didn't know about the new variables.
The 1st error seems to be because the computers weren't ready to talk after their failures, as I haven't seen that error after making the fixes. I still see it if I kill a worker and start him up again, but it doesn't seem to be an issue if they all start up together.
Anyway, I hope that's helpful if anyone ever has the same error later on.

iPython won't start anymore after using os.dup2()

I was just trying out the os.dup2() function to redirect outputs, when I was typing in os.dup2(3,1), which my ipython (2.7) didn't seem to like.
It crashed and now it won't start again, yielding the error:
Traceback (most recent call last):
File "/usr/bin/ipython", line 8, in <module>
launch_new_instance()
File "/usr/lib/python2.7/dist-packages/IPython/frontend/terminal/ipapp.py", line 402, in launch_new_instance
app.initialize()
File "<string>", line 2, in initialize
File "/usr/lib/python2.7/dist-packages/IPython/config/application.py", line 84, in catch_config_error
return method(app, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/IPython/frontend/terminal/ipapp.py", line 312, in initialize
self.init_shell()
File "/usr/lib/python2.7/dist-packages/IPython/frontend/terminal/ipapp.py", line 332, in init_shell
ipython_dir=self.ipython_dir)
File "/usr/lib/python2.7/dist-packages/IPython/config/configurable.py", line 318, in instance
inst = cls(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/IPython/frontend/terminal/interactiveshell.py", line 183, in __init__
user_module=user_module, custom_exceptions=custom_exceptions
File "/usr/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 456, in __init__
self.init_readline()
File "/usr/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 1777, in init_readline
self.refill_readline_hist()
File "/usr/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 1789, in refill_readline_hist
include_latest=True):
File "/usr/lib/python2.7/dist-packages/IPython/core/history.py", line 256, in get_tail
return reversed(list(cur))
DatabaseError: database disk image is malformed
If you suspect this is an IPython bug, please report it at:
https://github.com/ipython/ipython/issues
or send an email to the mailing list at ipython-dev#scipy.org
You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.
Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
c.Application.verbose_crash=True
can anyone help me with that?
Reposting as an answer:
That looks like fd 3 is your IPython history database, and you redirected stdout to it and corrupted it.
To get it to start again, remove or rename ~/.ipython/profile_default/history.sqlite (or ~/.config/ipython/profile_default/history.sqlite on certain IPython versions on Linux).