I was just trying out the os.dup2() function to redirect outputs, when I was typing in os.dup2(3,1), which my ipython (2.7) didn't seem to like.
It crashed and now it won't start again, yielding the error:
Traceback (most recent call last):
File "/usr/bin/ipython", line 8, in <module>
launch_new_instance()
File "/usr/lib/python2.7/dist-packages/IPython/frontend/terminal/ipapp.py", line 402, in launch_new_instance
app.initialize()
File "<string>", line 2, in initialize
File "/usr/lib/python2.7/dist-packages/IPython/config/application.py", line 84, in catch_config_error
return method(app, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/IPython/frontend/terminal/ipapp.py", line 312, in initialize
self.init_shell()
File "/usr/lib/python2.7/dist-packages/IPython/frontend/terminal/ipapp.py", line 332, in init_shell
ipython_dir=self.ipython_dir)
File "/usr/lib/python2.7/dist-packages/IPython/config/configurable.py", line 318, in instance
inst = cls(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/IPython/frontend/terminal/interactiveshell.py", line 183, in __init__
user_module=user_module, custom_exceptions=custom_exceptions
File "/usr/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 456, in __init__
self.init_readline()
File "/usr/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 1777, in init_readline
self.refill_readline_hist()
File "/usr/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 1789, in refill_readline_hist
include_latest=True):
File "/usr/lib/python2.7/dist-packages/IPython/core/history.py", line 256, in get_tail
return reversed(list(cur))
DatabaseError: database disk image is malformed
If you suspect this is an IPython bug, please report it at:
https://github.com/ipython/ipython/issues
or send an email to the mailing list at ipython-dev#scipy.org
You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.
Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
c.Application.verbose_crash=True
can anyone help me with that?
Reposting as an answer:
That looks like fd 3 is your IPython history database, and you redirected stdout to it and corrupted it.
To get it to start again, remove or rename ~/.ipython/profile_default/history.sqlite (or ~/.config/ipython/profile_default/history.sqlite on certain IPython versions on Linux).
Related
It takes nearly three hours for the simulator to get pass the initialization point when I try to run a PARSEC benchmark in full system mode, only to be met with the following output:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "build/X86/python/m5/main.py", line 457, in main
exec(filecode, scope)
File "configs/example/fs.py", line 396, in <module>
Simulation.run(options, root, test_sys, FutureClass)
File "configs/common/Simulation.py", line 726, in run
exit_event = benchCheckpoints(options, maxtick, cptdir)
File "configs/common/Simulation.py", line 269, in benchCheckpoints
exit_event = m5.simulate(maxtick - m5.curTick())
File "build/X86/python/m5/simulate.py", line 180, in simulate
return _m5.event.simulate(*args, **kwargs)
RuntimeError: bad_function_call
I don't even know where to begin solving this problem. Does anyone know what might be causing this issue and how to solve it? Thanks.
I have developed a SessionRunHook which I attach to TPUEstimator. SessionRunHook works perfectly fine on CPU, however if I use TPU I get an error:
INFO:tensorflow:Error recorded from outfeed: Attempted to use a closed Session.
INFO:tensorflow:Error recorded from evaluation_loop: Operation 'mean_1' has been marked as not fetchable.
INFO:tensorflow:evaluation_loop marked as finished
WARNING:tensorflow:Reraising captured error
Traceback (most recent call last):
File "/usr/lib/python3.5/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/donatas_repecka/workspace/bert/run_pretraining.py", line 549, in <module>
tf.app.run()
INFO:tensorflow:Error recorded from infeed: Step was cancelled by an explicit call to `Session::Close()`.
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "/home/donatas_repecka/workspace/bert/run_pretraining.py", line 505, in main
result = estimator.evaluate(input_fn=input_fn, steps=FLAGS.max_eval_steps)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py", line 2424, in evaluate
rendezvous.raise_errors()
File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/tpu/python/tpu/error_handling.py", line 128, in raise_errors
six.reraise(typ, value, traceback)
File "/usr/local/lib/python3.5/dist-packages/six.py", line 693, in reraise
raise value
File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/tpu/python/tpu/error_handling.py", line 101, in catch_errors
yield
File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py", line 451, in _run_outfeed
session.run(self._dequeue_ops)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1075, in _run
raise RuntimeError('Attempted to use a closed Session.')
RuntimeError: Attempted to use a closed Session.
Has anyone else experienced this problem and found the way around it?
I'm running a tensorflow model submitting the training on ml engine. I have built a pipeline which reads from BigQuery using tf.contrib.cloud.python.ops.bigquery_reader_ops.BigQueryReader as a reader for the queue.
Everything works fine using DataLab and in local, setting the GOOGLE_APPLICATION_CREDENTIALS variable pointing to the json file for the credentials key. However, when I submit the training job in the cloud I get these errors (I just post the main two):
Permission denied: Error executing an HTTP request (HTTP response code 403, error code 0, error message '') when reading schema for...
There was an error creating the model. Check the details: Request had insufficient authentication scopes.
I've already checked everything else like correctly defining the table schema in the script and project/dataset/table ids/names
I paste down here the whole error present in the log for more clarity:
message: "Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/root/.local/lib/python2.7/site-packages/trainer/task.py", line 131, in <module>
hparams=hparam.HParams(**args.__dict__)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/learn_runner.py", line 210, in run
return _execute_schedule(experiment, schedule)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/learn_runner.py", line 47, in _execute_schedule
return task()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 495, in train_and_evaluate
self.train(delay_secs=0)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 275, in train
hooks=self._train_monitors + extra_hooks)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 665, in _call_train
monitors=hooks)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 289, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 455, in fit
loss = self._train_model(input_fn=input_fn, hooks=hooks)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 1007, in _train_model
_, loss = mon_sess.run([model_fn_ops.train_op, model_fn_ops.loss])
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 521, in __exit__
self._close_internal(exception_type)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 556, in _close_internal
self._sess.close()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 791, in close
self._sess.close()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 888, in close
ignore_live_threads=True)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/queue_runner_impl.py", line 238, in _run
enqueue_callable()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1063, in _single_operation_run
target_list_as_strings, status, None)
File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
PermissionDeniedError: Error executing an HTTP request (HTTP response code 403, error code 0, error message '')
when reading schema for pasquinelli-bigdata:Transactions.t_11_Hotel_25_w_train#1505224768418
[[Node: GenerateBigQueryReaderPartitions = GenerateBigQueryReaderPartitions[columns=["F_RACC_GEST", "LABEL", "F_RCA", "W24", "ETA", "W22", "W23", "W20", "W21", "F_LEASING", "W2", "W16", "WLABEL", "SEX", "F_PIVA", "F_MUTUO", "Id_client", "F_ASS_VITA", "F_ASS_DANNI", "W19", "W18", "W17", "PROV", "W15", "W14", "W13", "W12", "W11", "W10", "W7", "W6", "W5", "W4", "W3", "F_FIN", "W1", "ImpTot", "F_MULTIB", "W9", "W8"], dataset_id="Transactions", num_partitions=1, project_id="pasquinelli-bigdata", table_id="t_11_Hotel_25_w_train", test_end_point="", timestamp_millis=1505224768418, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
Any suggestion would be extremely helpful since I'm relatively new with GC.
Thank you all.
Support for reading BigQuery data from Cloud ML Engine is still under development, so what you are doing is currently unsupported. The issue you are hitting is the machines that ML Engine runs do not have the right scopes to talk to BigQuery. A potential issue you may also encounter running locally is poor performance reading from BigQuery. These are two examples of work that needs to be addressed.
In the meantime, I recommend exporting data to GCS for training. This is going to be much more scalable so you don't have to worry about poor training performance as your data increases. This can be a good pattern as well as it will let you preprocess your data once, write the result to GCS in CSV format, and then do multiple training runs to try out different algorithms or hyperparameters.
I want to view statistics of my model by saving my graph to a file then running graph_metrics.py.
I have tried a few different things to write the file, my best effort is:
tf.train.write_graph( session.graph_def, ".", "my_graph", as_text=True )
But here's what happens:
$ python ./util/graph_metrics.py --noinput_binary --graph my_graph
Traceback (most recent call last):
File "./util/graph_metrics.py", line 137, in <module>
tf.app.run()
File ".virtualenv/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv))
File "./util/graph_metrics.py", line 85, in main
FLAGS.batch_size)
File "./util/graph_metrics.py", line 109, in calculate_graph_metrics
input_tensor = sess.graph.get_tensor_by_name(input_layer)
File ".virtualenv/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2531, in get_tensor_by_name
return self.as_graph_element(name, allow_tensor=True, allow_operation=False)
File ".virtualenv/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2385, in as_graph_element
return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
File ".virtualenv/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2427, in _as_graph_element_locked
"graph." % (repr(name), repr(op_name)))
KeyError: "The name 'Mul:0' refers to a Tensor which does not exist. The operation, 'Mul', does not exist in the graph."
Is there a complete working example of saving a graph, then analyzing it with graph_metrics.py?
This process seems to involve a magic incantation that I haven't yet discovered.
The error you're hitting is because you need to specify the name of your own input node with --input_layer= (it just defaults to Mul:0 because that's what we use in one of our Inception models):
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/tools/graph_metrics.py#L51
The graph_metrics script is still very much a work in progress unfortunately, and you may hit problems with shape inference, but hopefully this should get you past the initial hurdle.
Firstly, I know very very little about Python, Xwindows, Matplotlib or GTK. I am trying to run a tool called SpliceGrapher which uses the above. I get an (ugly, sorry) error:
/usr/lib64/python2.6/site-packages/gtk-2.0/gtk/__init__.py:57: GtkWarning: could not open display
warnings.warn(str(e), _gtk.Warning)
/home/my/bin/lib64/python2.6/site-packages/matplotlib/backends/backend_gtk.py:52: GtkWarning: gdk_cursor_new_for_display: assertion `GDK_IS_DISPLAY (display)' failed
cursors.MOVE : gdk.Cursor(gdk.FLEUR),
Traceback (most recent call last):
File "/home/my/bin/SpliceGrapher-0.2.0/scripts/view_splicegraphs.py", line 28, in <module>
from pylab import *
File "/home/my/bin/lib64/python2.6/site-packages/pylab.py", line 1, in <module>
from matplotlib.pylab import *
File "/home/my/bin/lib64/python2.6/site-packages/matplotlib/pylab.py", line 264, in <module>
from matplotlib.pyplot import *
File "/home/my/bin/lib64/python2.6/site-packages/matplotlib/pyplot.py", line 95, in <module>
new_figure_manager, draw_if_interactive, _show = pylab_setup()
File "/home/my/bin/lib64/python2.6/site-packages/matplotlib/backends/__init__.py", line 25, in pylab_setup
globals(),locals(),[backend_name])
File "/home/my/bin/lib64/python2.6/site-packages/matplotlib/backends/backend_gtkagg.py", line 10, in <module>
from matplotlib.backends.backend_gtk import gtk, FigureManagerGTK, FigureCanvasGTK,\
File "/home/my/bin/lib64/python2.6/site-packages/matplotlib/backends/backend_gtk.py", line 52, in <module>
cursors.MOVE : gdk.Cursor(gdk.FLEUR),
RuntimeError: could not create GdkCursor object
If anyone has any idea what I can do, that would be great.
I am on a Linux (RedHat) system, through puTTy using Xming. I have X11 configured and xeyes shows the little eyes that follow my mouse so I know thats working.
New error:
16:00:46 view_splicegraphs.py Started
Traceback (most recent call last):
File "/home/bmoran/bin/SpliceGrapher-0.2.0/scripts/view_splicegraphs.py", line 164, in <module>
g = getFirstGraph(f)
File "/home/bmoran/bin/lib64/python2.6/site-packages/SpliceGrapher/SpliceGraph.py", line 371, in getFirstGraph
result = SpliceGraphParser(f, **args).next()
File "/home/bmoran/bin/lib64/python2.6/site-packages/SpliceGrapher/SpliceGraph.py", line 1351, in __init__
self.loadFromFile()
File "/home/bmoran/bin/lib64/python2.6/site-packages/SpliceGrapher/SpliceGraph.py", line 1424, in loadFromFile
raise ValueError("Graph feature found before graph header at line %d" % lineNo)
ValueError: Graph feature found before graph header at line 1
/usr/lib64/python2.6/site-packages/gtk-2.0/gtk/__init__.py:57: GtkWarning: could not open display
Tells you your script does not have an access to your X session.
Can you post here your script?
Did you access the machine with ssh -XC ... ?
on the shell when you issue echo $DISPLAY what do you see?
When you add in your script print os.getenv("DISPLAY") do you see the same result?
one more note ... I don't know Xming but I can recommend you to try MobaXterm, it has a builtin ssh and X11 server compiled for Windows, and my guess is that it will solve your X problem.