I'm working with gym environment and stable-baseline3 library. I have wrapped the environment as follows,
env = AtariWrapper(gym.make(args.env), noop_max=30, frame_skip=4, screen_size=84, terminal_on_life_loss=True, clip_reward=False)
env = DummyVecEnv([lambda: env])
env = VecFrameStack(env, n_stack=4, channels_order='first')
Now, I woud like to get the RGB images of each steps. I have gone though the code blocks of the library, I have found there is function within DummyVecEnv class called get_images() which returns the rgb images. But when I called the function, I got the following error
TypeError: render() takes 1 positional argument but 2 were given
I can't change the order in which the gym environment is wrapped. Is there any method by which I can get the RGB images from the environment. Thanks in advance.
Related
I am working with Matplotlib color maps, and I also happen to be working with the Spyder IDE, which has an ipython console.
As you can see from the screen shot, the ipython console showed me a graphical representation of the color map object. This was unexpected and very helpful.
Normally I expect to see a string representation of an object, as you might see from the print() function call. Function calls to print() and repr() are shown, and they produce text, as is more typical.
I would like my own code to output this graphical representation when it is generating output. I have been poking through the matplotlib.colors.Colormap internals, and so far I haven't been able to figure out how. What is ipython doing? How can I do the same?
Thanks!
This rather seems like a ipython/jupyter feature. ipython detects the object and produces automatically a plot to preview the colormap.
Here using jupyter:
IPython looks if an object has a _repr_html_; if so, it calls it and displays the output as HTML. Here's an example (I ran this in Jupyter but it works the same as long as you're running IPython):
class MyCoolObject:
def _repr_html_(self):
return ("<h1>hello!</h1> <p>this is some html </p>"
"I can even put images:"
"<img src='https://upload.wikimedia.org/wikipedia"
"/commons/thumb/3/38/Jupyter_logo.svg"
"/44px-Jupyter_logo.svg.png'></img>")
MyCoolObject()
To add on to Eduardo's answer, from everything I've read adding a _repr_html_ method should make iPython display the object when you type it into the console. I also use spyder though, and could not get it to work the way I expected. This simple wrapper class should allow you to display any html:
class displayedHTML:
def __init__(self, html):
self.html = html
def _repr_html_(self):
return self.html
But as you can see it does not work for me, instead showing the (implicitly defined) __repr__ of the class.
In [2]: obj = displayedHTML("<h1>" + "hello world" + "</h1>")
In [3]: obj
Out[3]: <__main__.displayedHTML at 0x1e8cda8f0d0>
I was not able to find the reason why this does not work, but I found a workaround if you just want to display a matplotlib colormap in the console from code (like I did).
Since know the matplotlib object works correctly, we can just give it to the ipython display function:
from IPython.display import display #Included without import since IPython 5.4 and 6.1
viridis = matplotlib.cm.get_cmap('viridis')
display(viridis)
And for me this works...
not_allowed_to_insert_pictures_yet.jpg
Hope this helps!
I'm following the tutorial on using your own template images to do object 3D pose tracking, but I'm trying to get it working on Ubuntu 20.04 with a live webcam stream.
I was able to successfully make my index .pb file with extracted KNIFT features from my custom images.
It seems the next thing to do is load the provided template matching graph (in mediapipe/graphs/template_matching/template_matching_desktop.pbtxt) (replacing the index_proto_filename of the BoxDetectorCalculator with my own index file), and run it on a video input stream to track my custom object.
I was hoping that would be easiest to do in python, but am running into dependency problems.
(I installed mediapipe python with pip3 install mediapipe)
First, I couldn't find how to directly load a .pbtxt file as a graph in the mediapipe python API, but that's ok. I just load the text it contains and use that.
template_matching_graph_filepath=os.path.abspath("~/mediapipe/mediapipe/graphs/template_matching/template_matching_desktop.pbtxt")
graph = mp.CalculatorGraph(graph_config=open(template_matching_graph_filepath).read())
But I get missing calculator targets.
No registered object with name: OpenCvVideoDecoderCalculator; Unable to find Calculator "OpenCvVideoDecoderCalculator"
or
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/text_format.cc:309] Error parsing text-format mediapipe.CalculatorGraphConfig: 54:70: Could not find type "type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions" stored in google.protobuf.Any.
It seems similar to this troubleshooting case but, since I'm not trying to compile an application, I'm not sure how to link in the missing calculators.
How to I make the mediapipe python API aware of these graphs?
UPDATE:
I made decent progress by adding the graphs that the template_matching depends on to the cc_library deps of the mediapipe/python/BUILD file
cc_library(
name = "builtin_calculators",
deps = [
"//mediapipe/calculators/image:feature_detector_calculator",
"//mediapipe/calculators/image:image_properties_calculator",
"//mediapipe/calculators/video:opencv_video_decoder_calculator",
"//mediapipe/calculators/video:opencv_video_encoder_calculator",
"//mediapipe/calculators/video:box_detector_calculator",
"//mediapipe/calculators/tflite:tflite_inference_calculator",
"//mediapipe/calculators/tflite:tflite_tensors_to_floats_calculator",
"//mediapipe/calculators/util:timed_box_list_id_to_label_calculator",
"//mediapipe/calculators/util:timed_box_list_to_render_data_calculator",
"//mediapipe/calculators/util:landmarks_to_render_data_calculator",
"//mediapipe/calculators/util:annotation_overlay_calculator",
...
I also modified solution_base.py so it knows about BoxDetector's options.
from mediapipe.calculators.video import box_detector_calculator_pb2
...
CALCULATOR_TO_OPTIONS = {
'BoxDetectorCalculator':
box_detector_calculator_pb2
.BoxDetectorCalculatorOptions,
Then I rebuilt and installed mediapipe python from source with:
~/mediapipe$ python3 setup.py install --link-opencv
Then I was able to make my own class derived from SolutionBase
from mediapipe.python.solution_base import SolutionBase
class ObjectTracker(SolutionBase):
"""Process a video stream and output a video with edges of templates highlighted."""
def __init__(self,
object_knift_index_file_path):
super().__init__(binary_graph_path=object_pose_estimation_binary_file_path,
calculator_params={"BoxDetector.index_proto_filename": object_knift_index_file_path},
)
def process(self, image: np.ndarray) -> NamedTuple:
return super().process(input_data={'input_video':image})
ot = ObjectTracker(object_knift_index_file_path="/path/to/my/object_knift_index.pb")
Finally, I process a video frame from a cv2.VideoCapture
cv_video = cv2.VideoCapture(0)
result, frame = cv_video.read()
input_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
res = ot.process(image=input_frame)
So close! But I run into this error which I just don't know what to do with.
/usr/local/lib/python3.8/dist-packages/mediapipe/python/solution_base.py in process(self, input_data)
326 if data.shape[2] != RGB_CHANNELS:
327 raise ValueError('Input image must contain three channel rgb data.')
--> 328 self._graph.add_packet_to_input_stream(
329 stream=stream_name,
330 packet=self._make_packet(input_stream_type,
RuntimeError: Graph has errors:
Calculator::Open() for node "BoxDetector" failed: ; Error while reading file: /usr/local/lib/python3.8/dist-packages/
Looks like CalculatorNode::OpenNode() is trying to open the python API install path as a file. Maybe it has to do with the default_context. I have no idea where to go from here. :(
I created the minimum working example. You can reproduce it here
I created my own Initializer called ComplexGlorotUniform(Initializer).
Then I created a file like: init_dispatcher = {"complex_glorot_uniform": ComplexGlorotUniform()}
Finally, I did:
from tensorflow.keras.utils import get_custom_objects
get_custom_objects().update(init_dispatcher)
I generated a sequential model of Dense layers using kernel_initializer="complex_glorot_uniform".
Now when using tf.keras.models.clone_model I get the error:
ValueError: Unknown initializer: ComplexGlorotUniform. Please ensure this object is passed to the `custom_objects` argument. See https://www.tensorflow.org/guide/keras/save_and_serialize#registering_the_custom_object
I DO think the custom_object is working correctly because he knows it's ComplexGlorotUniform and not the string I gave. Also, the layer is created correctly, is when calling the clone model method that it gets broken.
Well, I am still unsure why but I solved it by changing the init_dispatcher to init_dispatcher = {"ComplexGlorotUniform": ComplexGlorotUniform} to make the name match the string.
I guess it is Ok as it works for me but I am unsure if this is how it is supposed to work.
I have a velocity field and a density profile defined by vx, vy, and vz, and dens. All of these variable are 3D arrays, e.g.
vx.shape
(256,256,256)
same for vy, vz, and dens.
The flow command works properly e.g.
fl = flow(vx,vy,vz, seedtype='point', integration_direction='both', scalars=dens)
and I can visualise the field line. Now, to retrieve the values of vx,vy,vz and dens at the positions x,y,z along the line what I did in the past (a few years ago with a previous version of mayavi, I suppose) was
dataset = fl.outputs[0]
then
x, y, z = dataset.points.to_array().T
densstream = np.array(dataset.point_data.get_array(0))
vstream = np.array(dataset.point_data.get_array(1))
But running
x, y, z = dataset.points.to_array().T
I get the following error:
AttributeError: 'AlgorithmOutput' object has no attribute 'points'.
Typing
dataset
I get
<tvtk.tvtk_classes.algorithm_output.AlgorithmOutput at 0x1132917d0>
I am using a mac OS 10.12.6, Python 2.7.15 and IPython 5.5.0.
Could you help me, please?
Cheers
Newer versions of mayavi have modified the output ports for some tvtk objects, from the older version's tvtk.obj.output into tvtk_obj.output_port. The latter is a algorithm_output subclass of tvtk and does not contain the point data, which is the reason for the error message.
In mayavi/modules/streamline.py, the method _streamline_type_changed(self, value):
updates the output port of the streamline object via configure_outputs(self, st), a method of tvtk/common.py, but the connection is directed to self.stream_tracer.output_port instead of self.stream_tracer.output. And the latter is the correct poly_data subclass and does contain the point data along the streamline.
So the cure is simply:
dataset = fl.stream_tracer.output
By the way, I've also tried modifying the function call into configure_outputs(self, st.output), although fl.outputs[0] is connected correctly to the poly_data output, yet the streamlines are no longer shown in the mayavi scene.
After following this tutorial on summaries and TensorBoard, I've been able to successfully save and look at data with TensorBoard. Is it possible to open this data with something other than TensorBoard?
By the way, my application is to do off-policy learning. I'm currently saving each state-action-reward tuple using SummaryWriter. I know I could manually store/train on this data, but I thought it'd be nice to use TensorFlow's built in logging features to store/load this data.
As of March 2017, the EventAccumulator tool has been moved from Tensorflow core to the Tensorboard Backend. You can still use it to extract data from Tensorboard log files as follows:
from tensorboard.backend.event_processing.event_accumulator import EventAccumulator
event_acc = EventAccumulator('/path/to/summary/folder')
event_acc.Reload()
# Show all tags in the log file
print(event_acc.Tags())
# E. g. get wall clock, number of steps and value for a scalar 'Accuracy'
w_times, step_nums, vals = zip(*event_acc.Scalars('Accuracy'))
Easy, the data can actually be exported to a .csv file within TensorBoard under the Events tab, which can e.g. be loaded in a Pandas dataframe in Python. Make sure you check the Data download links box.
For a more automated approach, check out the TensorBoard readme:
If you'd like to export data to visualize elsewhere (e.g. iPython
Notebook), that's possible too. You can directly depend on the
underlying classes that TensorBoard uses for loading data:
python/summary/event_accumulator.py (for loading data from a single
run) or python/summary/event_multiplexer.py (for loading data from
multiple runs, and keeping it organized). These classes load groups of
event files, discard data that was "orphaned" by TensorFlow crashes,
and organize the data by tag.
As another option, there is a script
(tensorboard/scripts/serialize_tensorboard.py) which will load a
logdir just like TensorBoard does, but write all of the data out to
disk as json instead of starting a server. This script is setup to
make "fake TensorBoard backends" for testing, so it is a bit rough
around the edges.
I think the data are encoded protobufs RecordReader format. To get serialized strings out of files you can use py_record_reader or build a graph with TFRecordReader op, and to deserialize those strings to protobuf use Event schema. If you get a working example, please update this q, since we seem to be missing documentation on this.
I did something along these lines for a previous project. As mentioned by others, the main ingredient is tensorflows event accumulator
from tensorflow.python.summary import event_accumulator as ea
acc = ea.EventAccumulator("folder/containing/summaries/")
acc.Reload()
# Print tags of contained entities, use these names to retrieve entities as below
print(acc.Tags())
# E. g. get all values and steps of a scalar called 'l2_loss'
xy_l2_loss = [(s.step, s.value) for s in acc.Scalars('l2_loss')]
# Retrieve images, e. g. first labeled as 'generator'
img = acc.Images('generator/image/0')
with open('img_{}.png'.format(img.step), 'wb') as f:
f.write(img.encoded_image_string)
You can also use the tf.train.summaryiterator: To extract events in a ./logs-Folder where only classic scalars lr, acc, loss, val_acc and val_loss are present you can use this GIST: tensorboard_to_csv.py
Chris Cundy's answer works well when you have less than 10000 data points in your tfevent file. However, when you have a large file with over 10000 data points, Tensorboard will automatically sampling them and only gives you at most 10000 points. It is a quite annoying underlying behavior as it is not well-documented. See https://github.com/tensorflow/tensorboard/blob/master/tensorboard/backend/event_processing/event_accumulator.py#L186.
To get around it and get all data points, a bit hacky way is to:
from tensorboard.backend.event_processing.event_accumulator import EventAccumulator
class FalseDict(object):
def __getitem__(self,key):
return 0
def __contains__(self, key):
return True
event_acc = EventAccumulator('path/to/your/tfevents',size_guidance=FalseDict())
It looks like for tb version >=2.3 you can streamline the process of converting your tb events to a pandas dataframe using tensorboard.data.experimental.ExperimentFromDev().
It requires you to upload your logs to TensorBoard.dev, though, which is public. There are plans to expand the capability to locally stored logs in the future.
https://www.tensorflow.org/tensorboard/dataframe_api
You can also use the EventFileLoader to iterate through a tensorboard file
from tensorboard.backend.event_processing.event_file_loader import EventFileLoader
for event in EventFileLoader('path/to/events.out.tfevents.xxx').Load():
print(event)
Surprisingly, the python package tb_parse has not been mentioned yet.
From documentation:
Installation:
pip install tensorflow # or tensorflow-cpu pip install -U tbparse # requires Python >= 3.7
Note: If you don't want to install TensorFlow, see Installing without TensorFlow.
We suggest using an additional virtual environment for parsing and plotting the tensorboard events. So no worries if your training code uses Python 3.6 or older versions.
Reading one or more event files with tbparse only requires 5 lines of code:
from tbparse import SummaryReader
log_dir = "<PATH_TO_EVENT_FILE_OR_DIRECTORY>"
reader = SummaryReader(log_dir)
df = reader.scalars
print(df)