I am manually trying to link embedding tensor with metadata.tsv, but I am getting following error: "$LOG_DIR/metadata.tsv is not a file."
I am running Tensorboard with following command :
tensorboard --logdir default/
and my projector_config.pbtxt file is following :
embeddings {
tensor_name: 'embedding/decoder_embedding_matrix'
metadata_path: '$LOG_DIR/metadata.tsv'
}
I have checked my log_dir and it has all files. [Screen shot attached]
LOG_DIR Screenshot:
Error Screenshot
It cannot recognize $LOG_DIR the way you have used it. Either edit projector_config.pbtxt manually to provide the full path, or use this in your code:
import os
embedding.metadata_path = os.path.join(LOG_DIR, 'metadata.tsv')
where again LOG_DIR should preferably be the full path (and not the relative one).
Related
I tried to visualize my training progress using tensorboard, but when I used the command for displaying tensorboard, nothing displayed and no error message. It just shows a blank page with the message "take too long to respond". This is the callback code and magic command that I used to display it.
#log directory
log_folder = "logs"
#callback
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_folder, histogram_freq=1)
#magic command
%tensorboard --logdir log_folder
and this is the image of the problem:
The Darknet guide to detect objects in images using pre-trained weights is here
The command to run is:
./darknet detect cfg/yolov3.cfg yolov3.weights data/dog.jpg
The result of the detection is currently saved in the current directory.
How can i change this directory where the output file predictions.jpg is saved?
it is under detector.c there is a function like this save_image(im, "predictions");
import subprocess
comand = "/darknet detector test cfg/coco.data cfg/yolov3.cfg yolov3.weights {0} -dont_show"
def __run_shell_command(command):
output = subprocess.check_output(command, shell=True).decode("ascii")
return output
images = ["a.jpg","b.jpg"]
for image in images:
__run_shell_command(command.format(image))
I am a newbie on the TensorFlow object detection library. I have a specific data set what I have to produce myself and labeled it with thousands of jpg. I have run the file to detect the object from these images.. any way.The end of the process i have gotten frozen_graph and from it I exported model.ckpl file to inference graph folder everything goes fine, and I have tested model.ckpl model on the object_detection.ipynb file it works fine. Until this step, there is no problem.
However,Am not able to understand how could convert that model.ckpl file to model.tflite file to use on android studio app.
I have see many things like but I am no idea what is the input_tensors = [...]
output_tensors = [...]
I may already know but what it was actually...
Could you show me how could I convert it?
Use tensorboard to find out your input and output layer. For reference follow these links -
https://heartbeat.fritz.ai/intro-to-machine-learning-on-android-how-to-convert-a-custom-model-to-tensorflow-lite-e07d2d9d50e3
Tensorflow Convert pb file to TFLITE using python
If you don't know your inputs and outputs, use summarize_graph tool and feed it your frozen model.
See command here
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/tools/graph_transforms#inspecting-graphs
If you have trained your model from scratch you must be having the .meta file. Also you need to specify the output node names using which you can create a .pb file. Please refer to below link on steps to create this file:
Tensorflow: How to convert .meta, .data and .index model files into one graph.pb file
Once this is created you can further convert your .pb to tflite as below:
import tensorflow as tf
graph_def_file = "model.pb"
input_arrays = ["model_inputs"]
output_arrays = ["model_outputs"]
converter = tf.lite.TFLiteConverter.from_frozen_graph(
graph_def_file, input_arrays, output_arrays)
tflite_model = converter.convert()
open("converted_model.tflite", "wb").write(tflite_model)
I ran this code snippet:
import os
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
from tensorflow.contrib.tensorboard.plugins import projector
LOG_DIR = 'logs'
metadata = os.path.join(LOG_DIR, 'metadata.tsv')
mnist = input_data.read_data_sets('MNIST_data')
input_1 = mnist.train.next_batch(10)
images = tf.Variable(input_1[0], name='images')
with open(metadata, 'w') as metadata_file:
for row in input_1[1]:
metadata_file.write('%d\n' % row)
with tf.Session() as sess:
saver = tf.train.Saver([images])
sess.run(images.initializer)
saver.save(sess, os.path.join(LOG_DIR, 'images.ckpt'))
config = projector.ProjectorConfig()
# One can add multiple embeddings.
# Link this tensor to its metadata file (e.g. labels).
embedding = config.embeddings.add()
embedding.tensor_name = images.name
embedding.metadata_path = metadata
# Saves a config file that TensorBoard will read during startup.
projector.visualize_embeddings(tf.summary.FileWriter(LOG_DIR), config)
And after this, I opened tensorboard embedding tab and it showed parsing metadata. However, it kept on loading that way endlessly. I tried another code and in that case, it kept loading on fetching spite Image. Is there something wrong with my tensorboard?
The problem is that TensorBoard couldn't find your metadata file, because it looks for the metadata file relative to the directory that you have specified with your '--logdir' parameter of the 'tensorboard' command.
So if you are opening TensorBoard with 'tensorboard --logdir logs', it will look for the metadata file in 'logs/logs/metadata.tsv'.
A possible fix for your code is to replace this line
embedding.metadata_path = metadata
with this one:
embedding.metadata_path = 'metadata.tsv'
In general, in order to debug errors TensorBoard you have to look at the response of the error messages in your browser console when looking at TensorBoard.
I'm using the tflearn wrapper over tensorflow to build a model, and would like to add metadata (labels) to the resultant embedding visualization. Is there a way to link a metadata.tsv file to a saved checkpoint after the fact of running it?
I've created a projector_config.pbtxt file in the logdir of the checkpoint summaries, with the metadata.tsv being in the same folder. The config looks like this:
embeddings {
tensor_name: "Embedding/W"
metadata_path: "C:/tmp/tflearn_logs/shallow_lstm/"
}
and was created using the code from the docs - https://www.tensorflow.org/how_tos/embedding_viz/
I've commented out the tf.Session part in the hopes of creating the metadata link without the need of doing so directly within a Session object, but I'm not sure if that's possible.
from tensorflow.contrib.tensorboard.plugins import projector
#with tf.Session() as sess:
config = projector.ProjectorConfig()
# One can add multiple embeddings.
embedding = config.embeddings.add()
embedding.tensor_name = 'Embedding/W'
# Link this tensor to its metadata file (e.g. labels).
embedding.metadata_path = 'C:/tmp/tflearn_logs/shallow_lstm/'
# Saves a config file that TensorBoard will read during startup.
projector.visualize_embeddings(tf.summary.FileWriter('/tmp/tflearn_logs/shallow_lstm/'), config)
Below is a snap of the current embedding visualization. Note the empty metadata. Is there a way to directly attach the desired metafile to this embedding?
I had the same problem and it is soloved now :)
Essentially, all you need to do is following 3 steps:
save model checkpoint, supposing ckeckpoint's directory is ckp_dir;
place projector_config.pbtxt and metadata.tsv in ckp_dir;
run tensorboard --logdir=ckp_dir and click the Embedding Tab
the content of projector_config.pbtxt is :
embeddings {
tensor_name: "embedding_name"
metadata_path: "metatdata.tsv"
}
This is the key to link the embedding to metadata.tsv. In tf.Session(), we often get the embedding's value like sess.run('embedding_name:0'). But in projector_config.pbtxt, we just type tensor_name: "embedding_name".
Generally, we can specify the checkpoint path and metadata_path in projector_config.pbtxt so that we can place checkpoint, projector_config.pbtxt and metadata.tsv in different directories. But i think it is too complicated. I just solved it as above.
the result shown here
Try this with your projector_config.pbtxt:
embeddings {
tensor_name: "Embedding/W"
metadata_path: "$LOGDIR/metadata.tsv"
}
Make sure your $LOGDIR is the same path you use to call tensorboard --logdir=$LOGDIR on your terminal; that is, it should be relative to your current directory (so it probably shouldn't include C:/..). Also include the filename in the metadata_path.
Let me know if this works for you, too.
I stumbled upon the same problem trying to display words instead of indices for the word2vec tutorial. To achieve that your projector_config.pbtxt should look like this:
embeddings {
tensor_name: "w_in"
metadata_path: "$LOGDIR/vocab.txt"
}
You might also want to modify the save_vocab function in the code linked above since, as is, it converts unicode to hex.
I'm having the same problem. [EDIT:] The way I can make it work, is by creating a subdir and putting all the checkpoint files there, using a full path to the metadata file. The 'trick' is that then if you give tensorboard --logdir that specific dir, it somehow parses the metadata and displays the words in the plot instead of just the indices. The downside is that then then embeddings aren't found anymore when you just use the base log dir as --logdir, so you always have to start a separate tensorboard instance to view the embeddings.
It's really quite annoying and I can't imagine this is the only way to make it work, but I spent hours before that one finally worked...