I making a numpy array of Drake expressions, and I am trying to type hint them correctly. This is what I am currently doing:
import numpy as np
import numpy.typing as npt
import pydrake.symbolic
def some_function(expr: pydrake.symbolic.expression) -> npt.NDArray[pydrake.symbolic.expression]:
return np.array([expr])
where I am using type hints as above. However, I am getting the following error message:
Could not specialize type "NDArray[ScalarType#NDArray]"
Type "Expression" cannot be assigned to type "generic"
"Expression" is incompatible with "generic"
The same error message shows for different Drake objects like pydrake.symbolic.Formula etc.
For the record, I know I can use npt.ArrayLike instead of npt.NDArray[...], and this does not throw an error. However, I would like to find a solution that shows explicitly what the contents of the numpy arrays will be.
I am trying to read text in a file Shakespear.txt line by line using
tensorflow TextLineDataset. Split the words in a line and write the words in another file txt.txt one word per line. Here is my code
import tensorflow as tf
tf.enable_eager_execution()
BATCH_SIZE=2
#from tensorflow.keras.model import Sequential
dataset_in_lines=tf.data.TextLineDataset("Shakespear.txt")
dataset=dataset_in_lines.map(lambda string: tf.string_split([string]).values)
with open("txt.txt","w") as f:
for k in dataset.take(2):
for x in k:
f.write("\n".join(x))
When i run it it gives the error: Cannot iterate over a scalar tensor
in the f.write line. Please help me figure out the issue
It will be helpful if you could share the shakespear.txt file, but based on your error, it seems liek it is receiving the tensor not the actual value.
So, you first need to get the value from tensor k, you can use k.numpy().
Replace for x in k: with for x in k.numpy():
Let us know if it works.
I found a better way, replace dataset=dataset_in_lines.map(lambda string:tf.string_split([string]).values) with tokenizer.tokenize. The following code achieves the objective(see https://www.tensorflow.org/tutorials/load_data/text for more details)
import tensorflow as tf
tf.enable_eager_execution()
import tensorflow_datasets as tfds
tokenizer = tfds.features.text.Tokenizer()
dataset_in_lines=tf.data.TextLineDataset("Shakespear.txt")
vocabulary_set = set()
for x in dataset_in_lines:
k=tokenizer.tokenize(x.numpy())
vocabulary_set.update(k)
with open("txt.txt","w") as f:
for x in vocabulary_set:
f.write(x+"\n")
I created a 3 dimensional object using numpy.random module such as
import numpy as np
b = np.random.randn(4,4,3)
Why can't we cast type float to b?
TypeError
actual code
You can't float(b) because b isn't a number, it's a multidimensional array/matrix. If you're trying to convert every element to a Python float, that's a bad idea because numpy numbers are more precise, but if you really want to do that for whatever reason, you can do b.tolist(), which returns a Python list of floats. However, I don't believe you can have a numpy matrix of native Python types because that doesn't make any sense.
I'm having trouble solving a discrepancy between something breaking at runtime, but using the exact same data and operations in the python console, having it work fine.
# f_err - currently has value 1.11819388872025
# l_scales - currently a numpy array [1.17840183376334 1.13456764589809]
sq_euc_dists = self.se_term(x1, x2, l_scales) # this is fine. It calls cdists on x1/l_scales, x2/l_scales vectors
return (f_err**2) * np.exp(-0.5 * sq_euc_dists) # <-- errors on this line
The error that I get is
AttributeError: 'Zero' object has no attribute 'exp'
However, calling those exact same lines, with the same f_err, l_scales, and x1, x2 in the console right after it errors out, somehow does not produce errors.
I was not able to find a post referring to the 'Zero' object error specifically, and the non-'Zero' ones I found didn't seem to apply to my case here.
EDIT: It was a bit lacking in info, so here's an actual (extracted) runnable example with sample data I took straight out of a failed run, which when run in isolation works fine/I can't reproduce the error except in runtime.
Note that the sqeucld_dist function below is quite bad and I should be using scipy's cdist instead. However, because I'm using sympy's symbols for matrix elementwise gradients with over 15 partial derivatives in my real data, cdist is not an option as it doesn't deal with arbitrary objects.
import numpy as np
def se_term(x1, x2, l):
return sqeucl_dist(x1/l, x2/l)
def sqeucl_dist(x, xs):
return np.sum([(i-j)**2 for i in x for j in xs], axis=1).reshape(x.shape[0], xs.shape[0])
x = np.array([[-0.29932052, 0.40997373], [0.40203481, 2.19895326], [-0.37679417, -1.11028267], [-2.53012051, 1.09819485], [0.59390005, 0.9735], [0.78276777, -1.18787904], [-0.9300892, 1.18802775], [0.44852545, -1.57954101], [1.33285028, -0.58594779], [0.7401607, 2.69842268], [-2.04258086, 0.43581565], [0.17353396, -1.34430191], [0.97214259, -1.29342284], [-0.11103534, -0.15112815], [0.41541759, -1.51803154], [-0.59852383, 0.78442389], [2.01323359, -0.85283772], [-0.14074266, -0.63457529], [-0.49504797, -1.06690869], [-0.18028754, -0.70835799], [-1.3794126, 0.20592016], [-0.49685373, -1.46109525], [-1.41276934, -0.66472598], [-1.44173868, 0.42678815], [0.64623684, 1.19927771], [-0.5945761, -0.10417961]])
f_err = 1.11466725760716
l = [1.18388412685279, 1.02290811104357]
result = (f_err**2) * np.exp(-0.5 * se_term(x, x, l)) # This runs fine, but fails with the exact same calls and data during runtime
Any help greatly appreciated!
Here is how to reproduce the error you are seeing:
import sympy
import numpy
zero = sympy.sympify('0')
numpy.exp(zero)
You will see the same exception you are seeing.
You can fix this (inefficiently) by changing your code to the following to make things floating point.
def sqeucl_dist(x, xs):
return np.sum([np.vectorize(float)(i-j)**2 for i in x for j in xs],
axis=1).reshape(x.shape[0], xs.shape[0])
It will be better to fix your gradient function using lambdify.
Here's an example of how lambdify can be used on partial d
from sympy.abc import x, y, z
expression = x**2 + sympy.sin(y) + z
derivatives = [expression.diff(var, 1) for var in [x, y, z]]
derivatives is now [2*x, cos(y), 1], a list of Sympy expressions. To create a function which will evaluate this numerically at a particular set of values, we use lambdify as follows (passing 'numpy' as an argument like that means to use numpy.cos rather than sympy.cos):
derivative_calc = sympy.lambdify((x, y, z), derivatives, 'numpy')
Now derivative_calc(1, 2, 3) will return [2, -0.41614683654714241, 1]. These are ints and numpy.float64s.
A side note: np.exp(M) will calculate the element-wise exponent of each of the elements of M. If you are trying to do a matrix exponential, you need np.linalg.exmp.
I am learning cython to speed up numpy. I wrote a code to see how to optimize numpy array calculation.
The python code is:
from numpy import *
def set_onsite(n):
a=linspace(0,n,n+1)
onsite=zeros([n+1,n+1],float)
for i in range(0,n+1):
onsite[i,i]=a[i]*a[i]
return onsite
Then, I tried to cythonize this code:
import numpy as np
cimport numpy as np
cimport cython
import cython
#cython.boundscheck(False)
#cython.wraparound(False)
#cython.nonecheck(False)
def set_onsite(np.int_t n):
cdef np.ndarray[double,ndim=1,mode='c'] a=np.linspace(0,n,n+1)
cdef np.ndarray[double,ndim=2,mode='c'] onsite=np.empty(n+1,n+1)
cdef np.int_t i
for i in range(0,n+1):
onsite[i,i]=a[i]*a[i]
return onsite
After running setup.py file, I got the .so file. I ran the code %timeit myfile.set_onsite(10000),but IPython showed
TypeError: data type not understood
So could anyone tell me what is going on here?
I checked my code many times but I did not figure out where the problem arises.
The problem has nothing to do with cython; it's just that np.empty expects the first argument to be the shape given as an int or tuple of ints. The second argument is interpreted as the dtype:
In [19]: np.empty(5,5)
TypeError: data type not understood
while np.empty((5,5)) returns an empty array of shape (5,5).
So instead use
cdef np.ndarray[double,ndim=2,mode='c'] onsite=np.empty((n+1,n+1))
Note the double set of parentheses around n+1, n+1. Or, use np.zeros instead of np.empty to make the Cython function match the Python function.
PS: When debugging Python, it is helpful to note not only the error message, but the line that raises the exception:
File "comp.pyx", line 13, in comp.set_onsite (comp.c:1290)
cdef np.ndarray[double,ndim=2,mode='c'] onsite=np.empty(n+1,n+1)
TypeError: data type not understood