complex h5 File to show - complex-numbers

I have a complex type h5 file. 2d array. I want to imshow it. But I have following error. What is wrong?
import h5py
import numpy as np
import matplotlib.pyplot as plt
with h5py.File('obj_0001.h5', 'r') as hdf:
ls = list(hdf.keys())
print('List of datasets in thies file: \n', ls)
data = hdf.get('dataset')
diff = np.array(data)
print('Shape of dataset: \n', diff.shape)
plt.figure(1)
plt.imshow(np.abs(diff))
plt.savefig('diff_test.png')
plt.show()
UFuncTypeError: ufunc 'absolute' did not contain a loop with signature matching types dtype([('real', '<f4'), ('imag', '<f4')]) -> dtype([('real', '<f4'), ('imag', '<f4')])

According to http://docs.h5py.org/en/stable/faq.html#what-datatypes-are-supported
h5py supports complex dtype, representing a HDF5 struc.
The error indicates diff.dtype is dtype([('real', '<f4'), ('imag', '<f4')]). I don't know if that is the result of your np.array(data) conversion or there's something different about how the data is stored on the file.
diff = data[:]
might be worth trying, since that's the preferred syntax for loading an array from a dataset.
But if diff is that structured array, you can make a complex dtype with:
In [303]: arr2 = np.ones((3,), np.dtype([('real','f'),('imag','f')]))
In [304]: arr2
Out[304]:
array([(1., 1.), (1., 1.), (1., 1.)],
dtype=[('real', '<f4'), ('imag', '<f4')])
In [305]: arr3 = arr2['real']+1j*arr2['imag']
In [306]: arr3
Out[306]: array([1.+1.j, 1.+1.j, 1.+1.j], dtype=complex64)
testing in abs:
In [307]: np.abs(arr2)
---------------------------------------------------------------------------
UFuncTypeError Traceback (most recent call last)
<ipython-input-307-333e28818b26> in <module>
----> 1 np.abs(arr2)
UFuncTypeError: ufunc 'absolute' did not contain a loop with signature matching types dtype([('real', '<f4'), ('imag', '<f4')]) -> dtype([('real', '<f4'), ('imag', '<f4')])
In [308]: np.abs(arr3)
Out[308]: array([1.4142135, 1.4142135, 1.4142135], dtype=float32)

Related

Can't populate matplotlib animation frame one point at a time

I'm currently trying to build an N-body simulation but I'm having a little trouble with plotting the results the way I'd like.
In the code below (with some example data for a few points in an orbit) I'm importing the position and time data and organizing it into a pandas dataframe. To create the 3D animation I use matplotlib's animation class, which works perfectly.
However, the usual way to set up an animation is limited in that you can't customize the points in each frame individually (please let me know if I'm wrong here :p). Since my animation is showing orbiting bodies I would like to vary their sizes and colors. To do that I essentially create a graph for each body and set it's color etc. When it gets to the update_graph function, I iterate over the n bodies, retrieve their individual (x,y,z) coordinates, and update their graphs.
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import cm
from mpl_toolkits.mplot3d.axes3d import get_test_data
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.animation as animation
import pandas as pd
nbodies = 2
x = np.array([[1.50000000e-10, 0.00000000e+00, 0.00000000e+00],
[9.99950000e-01, 1.00000000e-02, 0.00000000e+00],
[4.28093585e-06, 3.22964816e-06, 0.00000000e+00],
[-4.16142210e-01, 9.09335149e-01, 0.00000000e+00],
[5.10376489e-06, 1.42204430e-05, 0.00000000e+00],
[-6.53770813e-01, -7.56722445e-01, 0.00000000e+00]])
t = np.array([0.01, 0.01, 2.0, 2.0, 4.0, 4.0])
tt = np.array([0.01, 2.0, 4.0])
x = x.reshape((len(tt), nbodies, 3))
x_coords = x[:, :, 0].flatten()
y_coords = x[:, :, 1].flatten()
z_coords = x[:, :, 2].flatten()
df = pd.DataFrame({"time": t[:] ,"x" : x_coords, "y" : y_coords, "z" : z_coords})
print(df)
def update_graph(num):
data=df[df['time']==tt[num]] # x,y,z of all bodies at current time
for n in range(nbodies): # update graphs
data_n = data[data['x']==x_coords[int(num * nbodies) + n]] # x,y,z of body n
graph = graphs[n]
graph.set_data(data_n.x, data_n.y)
graph.set_3d_properties(data_n.z)
graphs[n] = graph
return graphs
plt.style.use('dark_background')
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.set_xlabel('x (AU)')
ax.set_ylabel('y (AU)')
ax.set_zlabel('z (AU)')
plt.xlim(-1.5,1.5)
plt.ylim(-1.5,1.5)
# initialize
data=df[df['time']==0]
ms_list = [5, 1]
c_list = ['yellow', 'blue']
graphs = []
for n in range(nbodies):
graphs.append(ax.plot([], [], [], linestyle="", marker=".",
markersize=ms_list[n], color=c_list[n])[0])
ani = animation.FuncAnimation(fig, update_graph, len(tt),
interval=400, blit=True, repeat=True)
plt.show()
However, doing this gives me the following error:
Traceback (most recent call last):
File "/home/kris/anaconda3/lib/python3.7/site-packages/matplotlib/backend_bases.py", line 1194, in _on_timer
ret = func(*args, **kwargs)
File "/home/kris/anaconda3/lib/python3.7/site-packages/matplotlib/animation.py", line 1447, in _step
still_going = Animation._step(self, *args)
File "/home/kris/anaconda3/lib/python3.7/site-packages/matplotlib/animation.py", line 1173, in _step
self._draw_next_frame(framedata, self._blit)
File "/home/kris/anaconda3/lib/python3.7/site-packages/matplotlib/animation.py", line 1193, in _draw_next_frame
self._post_draw(framedata, blit)
File "/home/kris/anaconda3/lib/python3.7/site-packages/matplotlib/animation.py", line 1216, in _post_draw
self._blit_draw(self._drawn_artists, self._blit_cache)
File "/home/kris/anaconda3/lib/python3.7/site-packages/matplotlib/animation.py", line 1231, in _blit_draw
a.axes.draw_artist(a)
File "/home/kris/anaconda3/lib/python3.7/site-packages/matplotlib/axes/_base.py", line 2661, in draw_artist
a.draw(self.figure._cachedRenderer)
File "/home/kris/anaconda3/lib/python3.7/site-packages/matplotlib/artist.py", line 38, in draw_wrapper
return draw(artist, renderer, *args, **kwargs)
File "/home/kris/anaconda3/lib/python3.7/site-packages/mpl_toolkits/mplot3d/art3d.py", line 202, in draw
xs, ys, zs = proj3d.proj_transform(xs3d, ys3d, zs3d, renderer.M)
File "/home/kris/anaconda3/lib/python3.7/site-packages/mpl_toolkits/mplot3d/proj3d.py", line 201, in proj_transform
vec = _vec_pad_ones(xs, ys, zs)
File "/home/kris/anaconda3/lib/python3.7/site-packages/mpl_toolkits/mplot3d/proj3d.py", line 189, in _vec_pad_ones
return np.array([xs, ys, zs, np.ones_like(xs)])
File "/home/kris/anaconda3/lib/python3.7/site-packages/pandas/core/series.py", line 871, in __getitem__
result = self.index.get_value(self, key)
File "/home/kris/anaconda3/lib/python3.7/site-packages/pandas/core/indexes/base.py", line 4405, in get_value
return self._engine.get_value(s, k, tz=getattr(series.dtype, "tz", None))
File "pandas/_libs/index.pyx", line 80, in pandas._libs.index.IndexEngine.get_value
File "pandas/_libs/index.pyx", line 90, in pandas._libs.index.IndexEngine.get_value
File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 997, in pandas._libs.hashtable.Int64HashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 1004, in pandas._libs.hashtable.Int64HashTable.get_item
KeyError: 0
Aborted (core dumped)
I'm not sure what this really means, but I do know the problem is something to do with updating the graphs with only one row of coordinates rather than all three. Because if I instead have
def update_graph(num):
data=df[df['time']==tt[num]] # x,y,z of all bodies at current time
for n in range(nbodies): # update graphs
#data_n = data[data['x']==x_coords[int(num * nbodies) + n]] # x,y,z of body n
graph = graphs[n]
graph.set_data(data.x, data.y) # using data rather than data_n here now
graph.set_3d_properties(data.z)
graphs[n] = graph
return graphs
it actually works, and plots three copies of the bodies with varying colors and sizes on top of each other as you would expect.
Any help would be much appreciated. Thanks!
I don't understand why you are going through a pandas DataFrame, when you seem to already have all the data you need in your numpy array. I couldn't reproduce the initial problem, by I propose this solution that uses pure numpy arrays, which may fix the problem:
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import cm
from mpl_toolkits.mplot3d.axes3d import get_test_data
from mpl_toolkits.mplot3d import Axes3D
import matplotlib.animation as animation
import pandas as pd
nbodies = 2
x = np.array([[1.50000000e-10, 0.00000000e+00, 0.00000000e+00],
[9.99950000e-01, 1.00000000e-02, 0.00000000e+00],
[4.28093585e-06, 3.22964816e-06, 0.00000000e+00],
[-4.16142210e-01, 9.09335149e-01, 0.00000000e+00],
[5.10376489e-06, 1.42204430e-05, 0.00000000e+00],
[-6.53770813e-01, -7.56722445e-01, 0.00000000e+00]])
t = np.array([0.01, 0.01, 2.0, 2.0, 4.0, 4.0])
tt = np.array([0.01, 2.0, 4.0])
x = x.reshape((len(tt), nbodies, 3))
def update_graph(i):
data = x[i, :, :] # x,y,z of all bodies at current time
for body, graph in zip(data, graphs): # update graphs
graph.set_data(body[0], body[1])
graph.set_3d_properties(body[2])
return graphs
plt.style.use('dark_background')
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.set_xlabel('x (AU)')
ax.set_ylabel('y (AU)')
ax.set_zlabel('z (AU)')
plt.xlim(-1.5, 1.5)
plt.ylim(-1.5, 1.5)
# initialize
ms_list = [50, 10]
c_list = ['yellow', 'blue']
graphs = []
for n in range(nbodies):
graphs.append(ax.plot([], [], [], linestyle="", marker=".",
markersize=ms_list[n], color=c_list[n])[0])
ani = animation.FuncAnimation(fig, func=update_graph, frames=len(tt),
interval=400, blit=True, repeat=True)
plt.show()

How to use a numpy array with fromiter

I tried to use a numpy array with fromiter but It gave this error
import numpy
l=numpy.dtype([("Ad","S20"),("Yas","i4"),("Derecelendirme","f")])
a=numpy.array([("Dr.Wah",20,0.9)])
d=numpy.fromiter(a,dtype=l,count=3)
print(d)
ValueError: setting an array element with a sequence.
In [172]: dt=np.dtype([("Ad","S20"),("Yas","i4"),("Derecelendirme","f")])
...: alist = [("Dr.Wah",20,0.9)]
The normal way to define a structured array is to use a list of tuples for the data along with the dtype:
In [173]: np.array( alist, dtype=dt)
Out[173]:
array([(b'Dr.Wah', 20, 0.9)],
dtype=[('Ad', 'S20'), ('Yas', '<i4'), ('Derecelendirme', '<f4')])
fromiter works as well, but isn't as common
In [174]: np.fromiter( alist, dtype=dt)
Out[174]:
array([(b'Dr.Wah', 20, 0.9)],
dtype=[('Ad', 'S20'), ('Yas', '<i4'), ('Derecelendirme', '<f4')])
If you create an array without the dtype:
In [175]: a = np.array(alist)
In [176]: a
Out[176]: array([['Dr.Wah', '20', '0.9']], dtype='<U6')
In [177]: _.shape
Out[177]: (1, 3)
a.astype(dt) does not work. You have to use a recfunction:
In [179]: import numpy.lib.recfunctions as rf
In [180]: rf.unstructured_to_structured(a, dtype=dt)
Out[180]:
array([(b'Dr.Wah', 20, 0.9)],
dtype=[('Ad', 'S20'), ('Yas', '<i4'), ('Derecelendirme', '<f4')])

AttributeError: 'Series' object has no attribute 'pipe'

I am getting the following error when I run the python3 keras code on my ec2 instance. The code works fine on Azure Jupyter Notebook.
Code:
numpy.random.seed(7)
dataframe = pandas.read_csv("some_data.csv", header = None)
df = dataframe
char_cols = df.dtypes.pipe(lambda x: x[x == 'object']).index
for c in char_cols:
df[c] = pandas.factorize(df[c])[0]
dataframe = df
Error:
Traceback (most recent call last):
File "pi_8_1st_year.py", line 12, in <module>
char_cols = df.dtypes.pipe(lambda x: x[x == 'object']).index
File "/usr/lib/python3/dist-packages/pandas/core/generic.py", line 1815, in __getattr__
(type(self).__name__, name))
AttributeError: 'Series' object has no attribute 'pipe'
My configuration:
ubuntu#ipxxxx:~$ python3
Python 3.4.3 (default, Nov 28 2017, 16:41:13)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
_frozen_importlib:321: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
/usr/lib/python3.4/importlib/_bootstrap.py:321: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
return f(*args, **kwds)
>>> pandas.__version__
'0.13.1'
>>> import numpy
>>> numpy.__version__
'1.15.0'
>>> import sklearn
/usr/lib/python3.4/importlib/_bootstrap.py:321: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
return f(*args, **kwds)
/usr/lib/python3.4/importlib/_bootstrap.py:321: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
return f(*args, **kwds)
>>> sklearn.__version__
'0.19.2'
>>> import keras
Using TensorFlow backend.
/usr/lib/python3.4/importlib/_bootstrap.py:321: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
return f(*args, **kwds)
>>> keras.__version__
'2.2.2'
>>>
As the error suggests you can't use pipe on series object. df.dtypes returns an series object, hence the error.
If you want to find columns of object types you can do it by:
s = (df.dtypes == 'object')
cols = s[s.values == True].index

while_loop in tensorflow returns type error

I am confused why the following code returns this error message:
Traceback (most recent call last):
File "/Users/Desktop/TestPython/tftest.py", line 46, in <module>
main(sys.argv[1:])
File "/Users/Desktop/TestPython/tftest.py", line 35, in main
result = tf.while_loop(Cond_f2, Body_f1, loop_vars=loopvars)
File "/Users/Desktop/HPC_LIB/TENSORFLOW/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2518, in while_loop
result = context.BuildLoop(cond, body, loop_vars, shape_invariants)
File "/Users/Desktop/HPC_LIB/TENSORFLOW/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2356, in BuildLoop
pred, body, original_loop_vars, loop_vars, shape_invariants)
File "/Users/Desktop/HPC_LIB/TENSORFLOW/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2292, in _BuildLoop
c = ops.convert_to_tensor(pred(*packed_vars))
File "/Users/Desktop/TestPython/tftest.py", line 18, in Cond_f2
boln = tf.less(tf.cast(tf.constant(ind), dtype=tf.int32), tf.cast(tf.constant(N), dtype=tf.int32))
File "/Users/Desktop/HPC_LIB/TENSORFLOW/lib/python2.7/site-packages/tensorflow/python/framework/constant_op.py", line 163, in constant
tensor_util.make_tensor_proto(value, dtype=dtype, shape=shape))
File "/Users/Desktop/HPC_LIB/TENSORFLOW/lib/python2.7/site-packages/tensorflow/python/framework/tensor_util.py", line 353, in make_tensor_proto
_AssertCompatible(values, dtype)
File "/Users/Desktop/HPC_LIB/TENSORFLOW/lib/python2.7/site-packages/tensorflow/python/framework/tensor_util.py", line 287, in _AssertCompatible
raise TypeError("List of Tensors when single Tensor expected")
TypeError: List of Tensors when single Tensor expected
I would appreciate if someone could help me fix this error. Thanks!
from math import *
import numpy as np
import sys
import tensorflow as tf
def Body_f1(n, ind, N, T):
# Compute trace
a = tf.trace(tf.random_normal(0.0, 1.0, (n, n)))
# Update trace
a = tf.cast(a, dtype=T.dtype)
T = tf.scatter_update(T, ind, a)
# Update index
ind = ind + 1
return n, ind, N, T
def Cond_f2(n, ind, N, T):
boln = tf.less(tf.cast(tf.constant(ind), dtype=tf.int32), tf.cast(tf.constant(N), dtype=tf.int32))
return boln
def main(argv):
# Open tensorflow session
sess = tf.Session()
# Parameters
N = 10
T = tf.zeros((N), dtype=tf.float64)
n = 4
ind = 0
# While loop
loopvars = [n, ind, N, T]
result = tf.while_loop(Cond_f2, Body_f1, loop_vars=loopvars, shape_invariants=None, \
parallel_iterations=1, back_prop=False, swap_memory=False, name=None)
trace = result[3]
trace = sess.run(trace)
print trace
print 'Done!'
# Close tensorflow session
if session==None:
sess.close()
if __name__ == "__main__":
main(sys.argv[1:])
Update: I have added the full error message. I am not sure why I get this error message. Does loop_vars expect a single tensor and not a list of tensors? I hope not.
tf.constant expects a non-Tensor value, like a Python list or a numpy array. You can get the same error by iterating tf.constant, as in tf.constant(tf.constant(5.)). Removing those calls fixes that first error. It's a very poor error message, so I would encourage you to file a bug on Github.
It also looks like the arguments to random_normal are a bit mixed up; keyword arguments are good for avoiding issues like that:
tf.random_normal(mean=0.0, stddev=1.0, shape=(n, n))
Finally scatter_update expects a variable. It looks like a TensorArray may be what you're looking for here (or one of the higher level looping constructs which use a TensorArray implicitly).

matplotlib.pyplot.imshow() shows blank canvas

I've come across an oddity that the internet hasn't been able to solve so far. If I read in a .png file, then try to show it, it works perfectly (in the example below the file is a single blue pixel). However, if I try to create this image array manually, it just shows a blank canvas. Any thoughts?
from PIL import Image
import matplotlib.pyplot as plt
import numpy as np
im = Image.open('dot.png') # A single blue pixel
im1 = np.asarray(im)
print im1
# [[[ 0 162 232 255]]]
plt.imshow(im1, interpolation='nearest')
plt.show() # Works fine
npArray = np.array([[[0, 162, 232, 255]]])
plt.imshow(npArray, interpolation='nearest')
plt.show() # Blank canvas
npArray = np.array([np.array([np.array([0, 162, 232, 255])])])
plt.imshow(npArray, interpolation='nearest')
plt.show() # Blank canvas
P.S. I've also tried replacing all of the np.array() with np.asarray(), but the outcome is just the same.
According to the im.show docs:
X : array_like, shape (n, m) or (n, m, 3) or (n, m, 4)
Display the image in `X` to current axes. `X` may be a float
array, a uint8 array or a PIL image.
So X may be an array with dtype uint8.
When you don't specify a dtype,
In [63]: np.array([[[0, 162, 232, 255]]]).dtype
Out[63]: dtype('int64')
NumPy may create an array of dtype int64 or int32 (not uint8) by default.
If you specify dtype='uint8' explicitly, then
import matplotlib.pyplot as plt
import numpy as np
npArray = np.array([[[0, 162, 232, 255]]], dtype='uint8')
plt.imshow(npArray, interpolation='nearest')
plt.show()
yields
PS. If you check
im = Image.open('dot.png') # A single blue pixel
im1 = np.asarray(im)
print(im1.dtype)
you'll find im1.dtype is uint8 too.