I am playing with the fixed code from this question. I am getting the above error. Googling suggests it might be some kind of dimension mismatch, though my diagnostics does not show any:
with tf.Session() as sess:
sess.run(init)
# Fit all training data
for epoch in range(training_epochs):
for (_x_, _y_) in getb(train_X, train_Y):
print("y data raw", _y_.shape )
_y_ = tf.reshape(_y_, [-1, 1])
print( "y data ", _y_.get_shape().as_list())
print("y place holder", yy.get_shape().as_list())
print("x data", _x_.shape )
print("x place holder", xx.get_shape().as_list() )
sess.run(optimizer, feed_dict={xx: _x_, yy: _y_})
Looking at the dimensions, everything is alright:
y data raw (20,)
y data [20, 1]
y place holder [20, 1]
x data (20, 10)
x place holder [20, 10]
Error:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-131-00e0bdc140b2> in <module>()
16 print("x place holder", xx.get_shape().as_list() )
17
---> 18 sess.run(optimizer, feed_dict={xx: _x_, yy: _y_})
19
20 # # Display logs per epoch step
/usr/local/lib/python3.4/dist-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict)
355 e.args = (e.message,)
356 raise e
--> 357 np_val = np.array(subfeed_val, dtype=subfeed_t.dtype.as_numpy_dtype)
358 if subfeed_t.op.type == 'Placeholder':
359 if not subfeed_t.get_shape().is_compatible_with(np_val.shape):
ValueError: setting an array element with a sequence.
Any debugging tips?
This—not very helpful—error is raised when one of the values in the feed_dict argument to tf.Session.run() is a tf.Tensor object (in this case, the result of tf.reshape()).
The values in feed_dict must be numpy arrays, or some value x that can be implicitly converted to a numpy array using numpy.array(x). tf.Tensor objects cannot be implicitly converted, because doing so might require a lot of work: instead you have to call sess.run(t) to convert a tensor t to a numpy array.
As you noticed in your answer, using np.reshape(_y_, [-1, 1]) works, because it produces a numpy array (and because _y_ is a numpy array to begin with). In general, you should always prepare data to be fed using numpy and other pure-Python operations.
replacing tf reshape with plain numpy one helped:
_y_ = np.reshape(_y_, [-1, 1])
the actual reason why is still unclear, but it works.
Related
Here's an example of behavior I cannot understand, maybe someone can share the insight into the logic behind it:
ccn = np.ones(1)
bbb = 7
bbn = np.array(bbb)
bbn * ccn # this is OK
array([7.])
np.prod((bbn,ccn)) # but this is NOT
Traceback (most recent call last):
File "C:\Program Files\JetBrains\PyCharm Community Edition 2020.2.2\plugins\python-ce\helpers\pydev\_pydevd_bundle\pydevd_exec2.py", line 3, in Exec
exec(exp, global_vars, local_vars)
File "<input>", line 1, in <module>
File "<__array_function__ internals>", line 5, in prod
File "C:\Users\...\venv\lib\site-packages\numpy\core\fromnumeric.py", line 2999, in prod
return _wrapreduction(a, np.multiply, 'prod', axis, dtype, out,
File "C:\Users\...\venv\lib\site-packages\numpy\core\fromnumeric.py", line 87, in _wrapreduction
return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
numpy.VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
Why? Why would a simple multiplication of two numbers be a problem? As far as formal algebra goes there's no dimensional problems, no datatype problems? The result is invariably also a single number, there's no chance it "suddenly" turn vector or object anything alike. prod(a,b) for a and b being scalars or 1by1 "matrices" is something MATLAB or Octave would eat no problem.
I know I can turn this error off and such, but why is it even and error?
In [346]: ccn = np.ones(1)
...: bbb = 7
...: bbn = np.array(bbb)
In [347]: ccn.shape
Out[347]: (1,)
In [348]: bbn.shape
Out[348]: ()
In [349]: np.array((bbn,ccn))
<ipython-input-349-997419ba7a2f>:1: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
np.array((bbn,ccn))
Out[349]: array([array(7), array([1.])], dtype=object)
You have arrays with different dimensions, that can't be combined into one numeric array.
That np.prod expression is actually:
np.multiply.reduce(np.array([bbn,ccn]))
can be deduced from your traceback.
In Octave both objects have shape (1,1), 2d
>> ccn = ones(1)
ccn = 1
>> ccn = ones(1);
>> size(ccn)
ans =
1 1
>> bbn = 7;
>> size(bbn)
ans =
1 1
>> [bbn,ccn]
ans =
7 1
It doesn't have true scalars; everything is 2d (even 3d is a fudge on the last dimension).
And with 'raw' Python inputs:
In [350]: np.array([1,[1]])
<ipython-input-350-f17372e1b22d>:1: VisibleDeprecationWarning: ...
np.array([1,[1]])
Out[350]: array([1, list([1])], dtype=object)
The object dtype array preserves the type of the inputs.
edit
prod isn't a simple multiplication. It's a reduction operation, like the big Pi in math. Even in Octave it isn't:
>> prod([[2,3],[3;4]])
error: horizontal dimensions mismatch (1x2 vs 2x1)
>> [2,3]*[3;4]
ans = 18
>> [2,3].*[3;4]
ans =
6 9
8 12
The numpy equivalent:
In [97]: np.prod((np.array([2,3]),np.array([[3],[4]])))
/usr/local/lib/python3.8/dist-packages/numpy/core/fromnumeric.py:87: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences...
return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
ValueError: could not broadcast input array from shape (2,1) into shape (2,)
In [98]: np.array([2,3])#np.array([[3],[4]])
Out[98]: array([18])
In [99]: np.array([2,3])*np.array([[3],[4]])
Out[99]:
array([[ 6, 9],
[ 8, 12]])
The warning, and here the error, is produced by trying to make ONE array from (np.array([2,3]),np.array([[3],[4]])).
Background
I am using Tensorflow for the first time following a tutorial on featurization with the new Google Recommenders package: https://www.tensorflow.org/recommenders/examples/featurization
I ran into trouble swapping out their dataset (MovieLens) for one based on the Kaggle wine data. The following code works as expected:
wine_title_lookup= tf.keras.layers.experimental.preprocessing.StringLookup()
wine_title_lookup.adapt(np.unique(wine_train['title']))
print(f"Vocabulary: {wine_title_lookup.get_vocabulary()[:3]}")
Vocabulary: ['', '[UNK]', 'Žitavské Vinice Rhine Riesling']
wine_title_embedding = tf.keras.layers.Embedding(
# Let's use the explicit vocabulary lookup.
input_dim=wine_title_lookup.vocab_size(),
output_dim=32
)
x= wine_title_lookup(["Susana Balbo Signature Malbec"])
x= wine_title_embedding(x)
x
<tf.Tensor: shape=(1, 32), dtype=float32, numpy=
array([[-0.03861505, -0.02146437, 0.04332292, -0.02598745, 0.03842534,
-0.01066433, 0.0292404 , 0.02783312, 0.03364438, 0.00054752,
-0.0295071 , 0.03200008, 0.01224083, -0.00100452, -0.04346857,
0.00105418, -0.01640136, -0.01778026, 0.00171928, 0.03215903,
0.00020416, -0.02083766, -0.00323264, 0.02582215, 0.04805436,
0.0325211 , 0.0100181 , -0.04965406, 0.02548517, 0.01569786,
0.03761304, 0.01659941]], dtype=float32)>
However the following produces an error
wine_title_model = tf.keras.Sequential([wine_title_lookup, wine_title_embedding])
wine_title_model(["Susana Balbo Signature Malbec"])
AttributeError Traceback (most recent call last)
in ()
----> 1 wine_title_model(["Susana Balbo Signature Malbec"])
3 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py in call(self, *args, **kwargs)
983
984 with ops.enable_auto_cast_variables(self._compute_dtype_object):
--> 985 outputs = call_fn(inputs, *args, **kwargs)
986
987 if self._activity_regularizer:
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/sequential.py in call(self, inputs, training, mask)
370 if not self.built:
371 self._init_graph_network(self.inputs, self.outputs)
--> 372 return super(Sequential, self).call(inputs, training=training, mask=mask)
373
374 outputs = inputs # handle the corner case where self.layers is empty
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/functional.py in call(self, inputs, training, mask)
384 """
385 return self._run_internal_graph(
--> 386 inputs, training=training, mask=mask)
387
388 def compute_output_shape(self, input_shape):
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/functional.py in _run_internal_graph(self, inputs, training, mask)
482 masks = self._flatten_to_reference_inputs(mask)
483 for input_t, mask in zip(inputs, masks):
--> 484 input_t._keras_mask = mask
485
486 # Dictionary mapping reference tensors to computed tensors.
AttributeError: 'str' object has no attribute '_keras_mask'
Notable differences with the source material
The Google code I based my script on uses a data format I am unfamiliar with which allows them to run map on their data. I tried converting my data into some tensorflow formats but could not seem to replicate their functionality. However this is the only step that is different and I cannot understand why the pieces of the Sequence op work individually but not as a whole.
I looked at some other examples from when this error has popped up on SO but could not find a solution to my problem. This what the raw data looks like.
wine_train.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 108655 entries, 0 to 120727
Data columns (total 16 columns):
Column Non-Null Count Dtype
--- ------ -------------- -----
0 country 108600 non-null object
1 description 108652 non-null object
2 designation 77150 non-null object
3 points 108336 non-null float64
4 price 100871 non-null float64
5 province 108600 non-null object
6 region_1 108655 non-null object
7 region_2 42442 non-null object
8 title 108655 non-null object
9 variety 108655 non-null object
10 winery 108655 non-null object
11 designation_replace 108655 non-null object
12 user_id 108655 non-null int64
13 price_isna 108655 non-null bool
14 price_imputed 108650 non-null float64
15 wine_id 108655 non-null int64
dtypes: bool(1), float64(3), int64(2), object(10)
memory usage: 13.4+ MB
Try using
wine_title_model.predict(["Susana Balbo Signature Malbec"])
I solved this problem by fixing the following areas of my code
Converting wine_train to a Tensorflow format
When I posted this question I had already tried running tf.data.Dataset.from_tensor_slices on my pandas dataframe. However it will not work. Instead convert the dataframe to a dictionary as so: wine_features_dict = {name: np.array(value) for name, value in wine_train.items()} and then everything runs smoothly.
Tensorflow is very sensitive to missing or NaN values.
I thought I got everything but just dropping all rows with any missing data seemed to get rid of the error. If it's happening to you make sure that there is no missing data and all your data is either integer or string.
Edited 13 Oct 2021:
Adding the full solution below - as requested
We want to convert data into a tf dictionary
wine_features_dict = {name: np.array(value) for name, value in wine_train.items()}
import itertools
def slices(features):
for i in itertools.count():
# For each feature take index `i`
example = {name:values[i] for name, values in features.items()}
yield example
for example in slices(wine_features_dict):
for name, value in example.items():
print(f"{name:19s}: {value}")
break
create features_ds
features_ds = tf.data.Dataset.from_tensor_slices(wine_features_dict)
for example in features_ds:
for name, value in example.items():
print(f"{name:19s}: {value}")
break
which yields
country : b'Portugal'
description : b"This is ripe and fruity, a wine that is smooth while still structured. Firm tannins are filled out with juicy red berry fruits and freshened with acidity. It's already drinkable, although it will certainly be better from 2016."
points : 87.0
price : 15.0
province : b'Douro'
title : b'Quinta dos Avidagos 2011 Avidagos Red (Douro)'
variety : b'Portuguese Red'
winery : b'Quinta dos Avidagos'
designation_replace: b'Avidagos'
user_id : b'15'
price_isna : False
price_imputed : 15.0
wine_id : 1
Preprocessing and other stuff
Between this and the actual model definition there are a bunch of things too lengthy for a SlackOverflow post. Essentially we are going to create embeddings for categorical variables and normalize or discretize continuous features. I added timestamps that I also had to preprocess. Word embeddings for text and then combine them
wine titles, user_ids -> vocabularies -> embeddings
price, price imputed -> normalize or discretize (bucket) -> embed buckets
words -> tokenization (splitting into constituent words or word-pieces), followed by vocabulary learning, followed by an embedding.
timestamps -> bucket based on max and min -> embed buckets
Next I explicitly defined the training inputs
inputs = {}
for name, column in wine_train.items():
dtype = column.dtype
if dtype == object:
dtype = tf.string
else:
dtype = tf.float32
inputs[name] = tf.keras.Input(shape=(1,), name=name, dtype=dtype)
and then the model
class UserModel(tf.keras.Model):
def __init__(self, use_timestamps=False, use_country_origin=False):
super().__init__()
self._use_timestamps = use_timestamps
self._use_country_origin = use_country_origin
self.user_embedding = tf.keras.Sequential([
tf.keras.layers.experimental.preprocessing.StringLookup(
vocabulary=unique_user_ids, mask_token=None),
tf.keras.layers.Embedding(len(unique_user_ids) + 1, 32),
])
'''
# Can also do this if user_id_lookup is defined
self.user_embedding = tf.keras.Sequential([
user_id_lookup,
tf.keras.layers.Embedding(user_id_lookup.vocab_size(), 32),
])
'''
if use_country_origin:
self.country_embedding = tf.keras.Sequential([
tf.keras.layers.experimental.preprocessing.StringLookup(
vocabulary=unique_countries, mask_token=None),
tf.keras.layers.Embedding(len(unique_countries) + 1, 32),
])
if use_timestamps:
self.timestamp_embedding = tf.keras.Sequential([
tf.keras.layers.experimental.preprocessing.Discretization(timestamp_buckets.tolist()),
tf.keras.layers.Embedding(len(timestamp_buckets) + 2, 32)
])
self.normalized_timestamp = tf.keras.layers.experimental.preprocessing.Normalization()
self.normalized_timestamp.adapt(timestamps)
def call(self, inputs):
# If timestamps not active just do the user_id embedding
if not self._use_timestamps:
# Ignore country of origin if is not enabled
if not self._use_country_origin:
return self.user_embedding(inputs['user_id'])
return tf.concat([
self.user_embedding(inputs["user_id"]),
self.country_embedding(inputs["country"]),
], axis=1)
# Take the input dictionary, pass it through each input layer,
# and concatenate the result.
if not self._use_country_origin:
return tf.concat([
self.user_embedding(inputs["user_id"]),
self.timestamp_embedding(inputs["timestamp"]),
self.normalized_timestamp(inputs["timestamp"]),
], axis=1)
return tf.concat([
self.user_embedding(inputs["user_id"]),
self.timestamp_embedding(inputs["timestamp"]),
self.normalized_timestamp(inputs["timestamp"]),
self.country_embedding(inputs["country"]),
], axis=1)
we can call it as well
user_model = UserModel()
# Delete quotes if timestamps are available
'''user_model.normalized_timestamp.adapt(
ratings.map(lambda x: x["timestamp"]).batch(128))
'''
for row in features_ds.batch(1).take(1):
print(f"Computed representations: {user_model(row)[0, :3]}")
I'm use python keras to build a cnn model.
I follow cnn mnist example and modify to my code.
This is the example I found
# Read MNIST data
(X_Train, y_Train), (X_Test, y_Test) = mnist.load_data()
# Translation of data
X_Train40 = X_Train.reshape(X_Train.shape[0], 28, 28, 1).astype('float32')
X_Test40 = X_Test.reshape(X_Test.shape[0], 28, 28, 1).astype('float32')
My data has 30222 rows and 6 columns of csv.
Which is 10074 data each data is 3 * 6 size for one block of information.
For example, the 1 ~ 3row of the matrix is one block of information.
Then I changed the format of my data.
X_Train40 = X_Train.reshape(10074, 3, 6, 1)
X_Test40 = X_Test.reshape(4319, 3, 6, 1)
Then this error occurs.
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-133-4f23172d450a> in <module>()
----> 1 X_Train40 = X_Train.reshape(10074, 3, 6, 1)
2 X_Test40 = X_Test.reshape(4319, 3, 6, 1)
~\Anaconda3\lib\site-packages\numpy\matrixlib\defmatrix.py in __array_finalize__(self, obj)
269 return
270 elif (ndim > 2):
--> 271 raise ValueError("shape too large to be a matrix.")
272 else:
273 newshape = self.shape
ValueError: shape too large to be a matrix.
Just guessing, but since the data comes from a csv file, it was converted to np.matrix, which have the restriction to be 2-dimensional.
Internally numpy will try to keep the dimensions of the matrix, so to reshape to higher dimensions, you will need to convert it to a ndarray like this:
X_Train = np.array(X_Train)
X_Test = np.array(X_Test)
X_Train40 = X_Train.reshape(10074, 3, 6, 1)
X_Test40 = X_Test.reshape(4319, 3, 6, 1)
You know... I had this problem today and I could not find a single answer that helped.
Your problem is probably solved at this point, but one thing to look at when python complains about "shape too large to be a matrix" is the type of your variable, namely, is it a numpy.matrix data type or a numpy.ndarray?
If it is the former, then you are in troubles.
Try to avoid numpy.matrix type, specially if you want to do any linear algebra operations, or stack them (with (d/v/h)stack, etc) and stick to numpy.ndarray
I'm trying to writting a layer to merge 2 tensors with such a formula
The shapes of x[0] and x[1] are both (?, 1, 500).
M is a 500*500 Matrix.
I want the output to be (?, 500, 500) which is theoretically feasible in my opinion. The layer will output (1,500,500) for every pair of inputs, as (1, 1, 500) and (1, 1, 500). As the batch_size is variable, or dynamic, the output must be (?, 500, 500).
However, I know little about axes and I have tried all the combinations of axes but it doesn't make sense.
I try with numpy.tensordot and keras.backend.batch_dot(TensorFlow). If the batch_size is fixed, taking a =
(100,1,500) for example, batch_dot(a,M,(2,0)), the output can be (100,1,500).
Newbie for Keras, sorry for such a stupid question but I have spent 2 days to figure out and it drove me crazy :(
def call(self,x):
input1 = x[0]
input2 = x[1]
#self.M is defined in build function
output = K.batch_dot(...)
return output
Update:
Sorry for being late. I try Daniel's answer with TensorFlow as Keras's backend and it still raises a ValueError for unequal dimensions.
I try the same code with Theano as backend and now it works.
>>> import numpy as np
>>> import keras.backend as K
Using Theano backend.
>>> from keras.layers import Input
>>> x1 = Input(shape=[1,500,])
>>> M = K.variable(np.ones([1,500,500]))
>>> firstMul = K.batch_dot(x1, M, axes=[1,2])
I don't know how to print tensors' shape in theano. It's definitely harder than tensorflow for me... However it works.
For that I scan 2 versions of codes for Tensorflow and Theano. Following are differences.
In this case, x = (?, 1, 500), y = (1, 500, 500), axes = [1, 2]
In tensorflow_backend:
return tf.matmul(x, y, adjoint_a=True, adjoint_b=True)
In theano_backend:
return T.batched_tensordot(x, y, axes=axes)
(If following changes of out._keras_shape don't make influence on out's value.)
Your multiplications should select which axes it uses in the batch dot function.
Axis 0 - the batch dimension, it's your ?
Axis 1 - the dimension you say has length 1
Axis 2 - the last dimension, of size 500
You won't change the batch dimension, so you will use batch_dot always with axes=[1,2]
But for that to work, you must ajust M to be (?, 500, 500).
For that define M not as (500,500), but as (1,500,500) instead, and repeat it in the first axis for the batch size:
import keras.backend as K
#Being M with shape (1,500,500), we repeat it.
BatchM = K.repeat_elements(x=M,rep=batch_size,axis=0)
#Not sure if repeating is really necessary, leaving M as (1,500,500) gives the same output shape at the end, but I haven't checked actual numbers for correctness, I believe it's totally ok.
#Now we can use batch dot properly:
firstMul = K.batch_dot(x[0], BatchM, axes=[1,2]) #will result in (?,500,500)
#we also need to transpose x[1]:
x1T = K.permute_dimensions(x[1],(0,2,1))
#and the second multiplication:
result = K.batch_dot(firstMul, x1T, axes=[1,2])
I prefer using TensorFlow so I tried to figure it out with TensorFlow in past few days.
The first one is much similar to Daniel's solution.
x = tf.placeholder('float32',shape=(None,1,3))
M = tf.placeholder('float32',shape=(None,3,3))
tf.matmul(x, M)
# return: <tf.Tensor 'MatMul_22:0' shape=(?, 1, 3) dtype=float32>
It needs to feed values to M with fit shapes.
sess = tf.Session()
sess.run(tf.matmul(x,M), feed_dict = {x: [[[1,2,3]]], M: [[[1,2,3],[0,1,0],[0,0,1]]]})
# return : array([[[ 1., 4., 6.]]], dtype=float32)
Another way is simple with tf.einsum.
x = tf.placeholder('float32',shape=(None,1,3))
M = tf.placeholder('float32',shape=(3,3))
tf.einsum('ijk,lm->ikl', x, M)
# return: <tf.Tensor 'MatMul_22:0' shape=(?, 1, 3) dtype=float32>
Let's feed some values.
sess.run(tf.einsum('ijk,kl->ijl', x, M), feed_dict = {x: [[[1,2,3]]], M: [[1,2,3],[0,1,0],[0,0,1]]})
# return: array([[[ 1., 4., 6.]]], dtype=float32)
Now M is a 2D tensor and no need to feed batch_size to M.
What's more, now it seems such a question can be solved in TensorFlow with tf.einsum. Does it mean it's a duty for Keras to invoke tf.einsum in some situations? At least I find no where Keras calls tf.einsum. And in my opinion, when batch_dot 3D tensor and 2D tensor Keras behaves weirdly. In Daniel's answer, he pads M to (1,500,500) but in K.batch_dot() M will be adjusted to (500,500,1) automatically. I find tf will adjust it with Broadcasting rules and I'm not sure Keras does the same.
If I want to slice after a scan operation in TensorFlow.
But I just get strange results with TensorFlow:
k = 10
x = 2
out = tf.scan(lambda previous_output, current_input: previous_output * current_input,
tf.fill([k], x), initializer=tf.constant(1))
result = out[-1] # slice with tensorflow - don't work
with tf.Session() as sess:
sess.run(tf.initialize_all_variables())
print(sess.run(out)[-1]) # works but all values are computed and stored in an np array
print(sess.run(result)) # don't work???
I get as output:
1024
3
The second value is obviously wrong and random (sometimes 0 or other values).
So my question is why? The analog code in Theano e.g. works and Theano can do some optimization when querying just the last element of the output tensor.