Related
I have an RGB image of shape (h, w, 3) and a corresponding depth map of shape (h, w).
Thus I know, for each pixel, its 3D coordinates.
I would like to rotate the image by some 3D rotation matrix.
I know how to apply the rotation to the input coordinates and get the coordinates in the target view, but how do I render the new view given the input image pixel values?
I tried using scipy's griddata, but this interpolation "fills" in gaps for occluded regions and overall performs interpolation, but not rendering of the new view.
Is there a better way to render the new rotated view in pytorch or numpy?
Here's some code that would do the association.
def get_colored_point_cloud(calib, rgb, depth):
"""
pass in rgb and associated depth map
return point cloud and color for each point
cloud.shape -> (num_points, 3) for [x, y, z]
colors.shape -> (num_points, 3) for [r, g, b]
"""
rows, cols = depth.shape
#create a grid and stack depth and rgb
c, r = np.meshgrid(np.arange(cols), np.arange(rows)) # c-> (cols, rows), r-> (cols, rows)
points = np.stack([c, r, depth]) # -> stacking (3, num_points)
colors = np.stack([c, r, rgb[:,:,0], rgb[:,:,1], rgb[:,:,2]])
points = points.reshape(3, -1) #-> (3, num_points)
colors = colors.reshape(5, -1) #-> (5, num_points)
points = points.T #-> (num_points, 3)
colors = colors.T #-> (num_points, 5)
#now transform [u, v] to [x, y, z] by camera unprojection
cloud = unproject_image_to_point_cloud(points, calib.intrinsic_params) #-> (num_points, 3)
return cloud, colors[:,2:5] # (num_points, 3), (num_points, 3)
It is also possible to do this through open3d. But you will have to deal with practical matters of getting the view as desired for it to work in open3d.
See this post: Generate point cloud from depth image
The more direct way of doing this instead of the somewhat ugly meshgrid process (at least the way I have written it) is by creating separate arrays for point(col_index, row_index, z) and color(col_index, row_index, R, G, B), and transforming (col_index, row_index,z) to (x, y, z) in an unrolled way for each point, but this is much slower as it does not use numpy vectorization magic under the hood.
def get_colored_point_cloud(calib, rgb, depth):
points = []
colors = []
rows, cols = depth.shape
for i in range(rows):
for j in range(cols):
z = depth[i, j]
r = rgb[i,j,0]
g = rgb[i,j,1]
b = rgb[i,j,2]
points.append([j,i,z])
colors.append([r,g,b])
points = np.asarray(points)
colors = np.asarray(colors)
cloud = unproject_image_to_point_cloud(points,\
calib.intrinsic_params) #-> (num_points, 3)
return cloud
I am trying to create an image from a matrix z2 over a raster defined by np.meshgrid(grid_x, grid_y) such that the value of the image at vx=grid_x[i], vy=grid_y[j] is z2[i, j]. On top of this image, I am trying to add a scatter plot of a number of points obtained by three vectors x, y, z such that the i-th point has the coordinate (x[k], y[k]) and the value z[k]. All of these scattered points lies within the region of the aforementioned raster.
Here's an example of the aforementioned data I am trying to plot.
import numpy as np
np.random.seed(1)
z2 = np.ones((1000, 1000)) * 0.66
z2[0, 0] = 0
z2[-1, -1] = 1
x = np.random.rand(1000) * 1000
y = np.random.rand(1000) * 1000
z = np.random.rand(1000)
grid_x = np.linspace(0, 999, 1000)
grid_y = np.linspace(0, 999, 1000)
In order to do this, I am using a 2D plot where the x and y values are used to define the position of the points and z is indicated by a color drawn from a colormap.
What is required of this image is that 1) there should be no white space between the actual plot and the edge of the figure; 2) the unit length on the x and y axis should be equal; 3) the image should not be too large. In order to achieve these, I am using the following code for plotting.
import matplotlib.pyplot as plt
from matplotlib import cm
def plot_img(x, y, z, grid_x, grid_y, z2, set_fig_size=True):
# determine the figure size
if set_fig_size:
height, width = np.array(z2.shape, dtype=float)
dpi = max(max(640 // height, 640 // width), 1)
width, height = width * dpi, height * dpi
plt.gcf().set_size_inches(width, height)
plt.gcf().set_dpi(dpi)
# plot the figure
plt.gca().axis('off')
plt.gca().axis('equal')
plt.gca().set_position([0, 0, 1, 1])
plt.xlim((grid_x[0], grid_x[-1]))
plt.ylim((grid_y[0], grid_y[-1]))
# the raster
cmap = cm.get_cmap('gray')
cmap.set_bad(color='red', alpha=0.5)
plt.imshow(z2, cmap=cmap, interpolation='none', origin='lower',
extent=(grid_x[0], grid_x[-1], grid_y[0], grid_y[-1]))
# the scatter plot
min_z, max_z = np.min(z), np.max(z)
c = (z - min_z) / (max_z - min_z)
plt.scatter(x, y, marker='o', c=c, cmap='Greens')
plt.show()
Strangely, when I run plot_img(x, y, z, grid_x, grid_y, z2) using the aforementioned example data, the following image shows up.
Essentially, only the raster data got plotted, while the scattered data is not.
I then tried plot_img(x, y, z, grid_x, grid_y, z2, set_fig_size=False). The result is
Note that here to clearly show the white spaces in the figure, I kept the background of PyCharm surrounding it. Essentially, there are white spaces that I do not wish included in this figure.
I wonder why this is happening, and how I can fix the code to get the correct output, which is essentially the second result without the white spaces. Thanks!
Replace your dpi and figsize code by
# determine the figure size
height, width = np.array(z2.shape, dtype=float)
dpi = 200
# get size in inches:
width, height = height / dpi, width / dpi
plt.gcf().set_size_inches(width, height)
plt.gcf().set_dpi(dpi)
and you will have a 1000x1000 pixel figure, which at 200 dpi is 5"x5".
I have a map figure rendered with Cartopy and Matplotlib. I have a specific geometric coordinate (in lat/lon) and I would like to know the pixel coordinate closest to this geometric coordinate's projection (if it is visible), for instance to draw a graphic over the coordinate on the map.
(Note I don't want to draw with Matplotlib; I'm exporting the figure as a bitmap image and drawing in a different part of the pipeline.)
This documentation suggests it might be something like this:
import cartopy, matplotlib.pyplot
fig = matplotlib.pyplot.figure()
ax = fig.add_axes([0, 0, 1, 1], projection=cartopy.crs.Orthographic())
ax.add_feature(cartopy.feature.LAND, facecolor='black')
# Print the location of New York City in display coordinates
lon, lat = -74.0060, 40.7128
trans = cartopy.crs.Geodetic()._as_mpl_transform(ax)
x, y = trans.transform((lon, lat))
print(x, y)
# Or this way
projx, projy = ax.projection.transform_point(lon, lat, cartopy.crs.Geodetic())
x, y = ax.transData.transform((projx, projy))
print(x, y)
Though interestingly, if I plot this point, the figure centers on and zooms into Manhattan, and then the output display coordinates are indeed in the center of the figure at (640, 480).
matplotlib.pyplot.plot(lon, lat, marker='o', color='red', markersize=12,
alpha=0.7, transform=cartopy.crs.Geodetic())
I just found that the transforms are not properly set until the figure is in its final state. So the key is to first draw the figure,
fig.canvas.draw()
or at least apply the aspect properly.
ax.apply_aspect()
Then you will get the correct pixel coordinates out,
import matplotlib.pyplot as plt
import cartopy
import cartopy.crs as ccrs
fig = plt.figure()
ax = fig.add_axes([0, 0, 1, 1], projection=ccrs.PlateCarree())
ax.add_feature(cartopy.feature.LAND, facecolor='black')
ax.set_global()
# before being able to call any of the transforms, the figure needs to be drawn
fig.canvas.draw()
# or
# ax.apply_aspect()
# Print the location of New York City in display coordinates
lon, lat = -74.0060, 40.7128
trans = ccrs.PlateCarree()._as_mpl_transform(ax)
x, y = trans.transform_point((lon, lat))
print(x,y)
plt.show()
This prints:
188.43377777777778 312.3783111111111
Note that those coordinates refer to the pixels from the lower left corner.
In my sample code I failed to specify the extent of the map. If I add
ax.set_global()
then the transformed coordinates are sensible.
I presented two ways to compute the transformed coordinates, but the way with _as_mpl_transform() seems to return the center point when New York City is not visible. The way with ax.projection.transform_point() returns NaN when off-screen.
I'm working on the ROI pooling layer which work for fast-rcnn and I am used to use tensorflow. I found tf.image.crop_and_resize can act as the ROI pooling layer.
But I try many times and cannot get the result that I expected.Or did the true result is exactly what I got?
here is my code
import cv2
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
img_path = r'F:\IMG_0016.JPG'
img = cv2.imread(img_path)
img = img.reshape([1,580,580,3])
img = img.astype(np.float32)
#img = np.concatenate([img,img],axis=0)
img_ = tf.Variable(img) # img shape is [580,580,3]
boxes = tf.Variable([[100,100,300,300],[0.5,0.1,0.9,0.5]])
box_ind = tf.Variable([0,0])
crop_size = tf.Variable([100,100])
#b = tf.image.crop_and_resize(img,[[0.5,0.1,0.9,0.5]],[0],[50,50])
c = tf.image.crop_and_resize(img_,boxes,box_ind,crop_size)
sess = tf.Session()
sess.run(tf.global_variables_initializer())
a = c.eval(session=sess)
plt.imshow(a[0])
plt.imshow(a[1])
And I handed in my origin img and result:a0,a1
if I was wrong can anyone teach me how to use this function? thanks.
Actually, there's no problem with Tensorflow here.
From the doc of tf.image.crop_and_resize (emphasis is mine) :
boxes: A Tensor of type float32. A 2-D tensor of shape [num_boxes, 4].
The i-th row of the tensor specifies the coordinates of a box in the
box_ind[i] image and is specified in normalized coordinates [y1, x1,
y2, x2]. A normalized coordinate value of y is mapped to the image
coordinate at y * (image_height - 1), so as the [0, 1] interval of
normalized image height is mapped to [0, image_height - 1] in image
height coordinates. We do allow y1 > y2, in which case the sampled
crop is an up-down flipped version of the original image. The width
dimension is treated similarly. Normalized coordinates outside the [0,
1] range are allowed, in which case we use extrapolation_value to
extrapolate the input image values.
The boxes argument needs normalized coordinates. That's why you get a black box with your first set of coordinates [100,100,300,300] (not normalized, and no extrapolation value provided), and not with your second set [0.5,0.1,0.9,0.5].
However, as that why matplotlib show you gibberish on your second attempt, it's just because you're using the wrong datatype.
Quoting the matplotlib documentation of plt.imshow (emphasis is mine):
All values should be in the range [0 .. 1] for floats or [0 .. 255]
for integers. Out-of-range values will be clipped to these bounds.
As you're using float outside the [0,1] range, matplotlib is bounding your values to 1. That's why you get those colored pixels (either solid red, solid green or solid blue, or a mixing of these). Cast your array to uint_8 to get an image that make sense.
plt.imshow( a[1].astype(np.uint8))
Edit :
As requested, I will dive a bit more into
tf.image.crop_and_resize.
[When providing non normalized coordinates and no extrapolation values], why I just get a blank result?
Quoting the doc :
Normalized coordinates outside the [0, 1] range are allowed, in which
case we use extrapolation_value to extrapolate the input image values.
So, normalized coordinates outside [0,1] are allowed. But they still need to be normalized !
With your example, [100,100,300,300], the coordinates you provide makes the red square. Your original image is the little green dot in the upper left corner! The default value of the argument extrapolation_value is 0, so the values outside the frame of the original image are inferred as [0,0,0] hence the black.
But if your usecase needs another value, you can provide it. The pixels will take a RGB value of extrapolation_value%256 on each channel. This option is useful if the zone you need to crop is not fully included in you original images. (A possible usecase would be sliding windows for example).
It seems that tf.image.crop_and_resize expects pixel values in the range [0,1].
Changing your code to
test = tf.image.crop_and_resize(image=image_np_expanded/255., ...)
solved the problem for me.
Yet another variant is to use tf.central_crop function.
Below is a concrete implementation of the tf.image.crop_and_resize API. tf version 1.14
import tensorflow as tf
import matplotlib.image as mpimg
import matplotlib.pyplot as plt
import numpy as np
tf.enable_eager_execution()
def single_data_2(img_path):
img = tf.read_file(img_path)
img = tf.image.decode_bmp(img,channels=1)
img_4d = tf.expand_dims(img, axis=0)
processed_img = tf.image.crop_and_resize(img_4d,boxes=
[[0.4529,0.72,0.4664,0.7358]],crop_size=[64,64],box_ind=[0])
processed_img_2 = tf.squeeze(processed_img,0)
raw_img_3 = tf.squeeze(img_4d,0)
return raw_img_3, processed_img_2
def plot_two_image(raw,processed):
fig=plt.figure(figsize=(35,35))
raw_ = fig.add_subplot(1,2,1)
raw_.set_title('Raw Image')
raw_.imshow(raw,cmap='gray')
processed_ = fig.add_subplot(1,2,2)
processed_.set_title('Processed Image')
processed_.imshow(processed,cmap='gray')
img_path = 'D:/samples/your_bmp_image.bmp'
raw_img, process_img = single_data_2(img_path)
print(raw_img.dtype,process_img.dtype)
print(raw_img.shape,process_img.shape)
raw_img=tf.squeeze(raw_img,-1)
process_img=tf.squeeze(process_img,-1)
print(raw_img.dtype,process_img.dtype)
print(raw_img.shape,process_img.shape)
plot_two_image(raw_img,process_img)
Below is my working code, also output image is not black, this can be of help to someone
for idx in range(len(bboxes)):
if bscores[idx] >= Threshold:
#Region of Interest
y_min = int(bboxes[idx][0] * im_height)
x_min = int(bboxes[idx][1] * im_width)
y_max = int(bboxes[idx][2] * im_height)
x_max = int(bboxes[idx][3] * im_width)
class_label = category_index[int(bclasses[idx])]['name']
class_labels.append(class_label)
bbox.append([x_min, y_min, x_max, y_max, class_label, float(bscores[idx])])
#Crop Image - Working Code
cropped_image = tf.image.crop_to_bounding_box(image, y_min, x_min, y_max - y_min, x_max - x_min).numpy().astype(np.int32)
# encode_jpeg encodes a tensor of type uint8 to string
output_image = tf.image.encode_jpeg(cropped_image)
# decode_jpeg decodes the string tensor to a tensor of type uint8
#output_image = tf.image.decode_jpeg(output_image)
score = bscores[idx] * 100
file_name = tf.constant(OUTPUT_PATH+image_name[:-4]+'_'+str(idx)+'_'+class_label+'_'+str(round(score))+'%'+'_'+os.path.splitext(image_name)[1])
writefile = tf.io.write_file(file_name, output_image)
I am able to make a plot of data points based on their Lat and Long, which looks like:
whereby the orange is made up of points like:
using the code:
m = Basemap(projection='merc',llcrnrlat=-0.5,urcrnrlat=0.5,\
llcrnrlon=9,urcrnrlon=10,lat_ts=0.25,resolution='i')
m.drawcoastlines()
m.drawcountries()
# draw parallels and meridians.
parallels = np.arange(-9.,10.,0.5)
# Label the meridians and parallels
m.drawparallels(parallels,labels=[False,True,True,False])
# Draw Meridians and Labels
meridians = np.arange(-1.,1.,0.5)
m.drawmeridians(meridians,labels=[True,False,False,True])
m.drawmapboundary(fill_color='white')
x,y = m(X, Y) # This is the step that transforms the data into the map's projection
scatter = plt.scatter(x,y)
m.scatter(x,y)
where X and Y are numpy arrays.
I want to get the X and Y co-ordinate of a point that I click on.
I can get the co-ord using:
coords = []
def onclick(event):
if plt.get_current_fig_manager().toolbar.mode != '':
return
global coords
ix, iy = event.x, event.y
print('x = %d, y = %d'%(ix, iy))
global coords
coords.append((ix, iy))
return coords
cid = fig.canvas.mpl_connect('button_press_event', onclick)
plt.show()
but this seems to return the figure co-ordinates. Is there a way to convert these to their respective lat and long co-ordinates?
I then plan to use these to find the nearest point in the original X and Y arrays to where I click
First of all you would want to use the data coordinates
ix, iy = event.xdata, event.ydata
Then to get lon/lat coordinates you need to apply the inverse map transform
lon, lat = m(event.xdata, event.ydata, inverse=True)