I am using PTB dataset to predict next words.
My code : pastebin link.
The input to the model (Batch_input) are the words with vocabulary_size of 10000. All the outputs (Batch_labels) are one-hot encoded as you can see a sample in the part of output code below.
Following is my output after training the LSTM Model.
Output : pastebin link.
Following is some part of output :
Initialized
('Loss :', 9.2027139663696289)
('Batch_input :', array([9971, 9972, 9974, 9975, 9976, 9980, 9981, 9982, 9983, 9984, 9986,
9987, 9988, 9989, 9991, 9992, 9993, 9994, 9995, 9996, 9997, 9998,
9999, 2, 9256, 1, 3, 72, 393, 33, 2133, 0, 146,
19, 6, 9207, 276, 407, 3, 2, 23, 1, 13, 141,
4, 1, 5465, 0, 3081, 1596, 96, 2, 7682, 1, 3,
72, 393, 8, 337, 141, 4, 2477, 657, 2170], dtype=int32))
('Batch_labels :', array([[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
...,
[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.]], dtype=float32))
Average loss at step 0: 0.092027 learning rate: 1.000000
('Label: ', array([[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
...,
[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.]], dtype=float32))
('Predicted:', array([[-0.36508381, -0.25612 , -0.26035795, ..., -0.42688274,
-0.4078168 , -0.36345699],
[-0.46035308, -0.27282876, -0.34078932, ..., -0.50623679,
-0.47014061, -0.43237451],
[-0.14694197, -0.07506246, -0.10392818, ..., -0.1128526 ,
-0.12404554, -0.13495158],
...,
[-0.07286638, -0.04560997, -0.05932444, ..., -0.08352474,
-0.07679331, -0.07829094],
[-0.13576414, -0.07057529, -0.1017022 , ..., -0.11192483,
-0.14713599, -0.11757012],
[-0.05446544, -0.02738103, -0.03401792, ..., -0.05073205,
-0.03746928, -0.05750648]], dtype=float32))
================================================================================
[[ 0. 0. 0. ..., 0. 0. 0.]]
8605
('f', u'altman')
('as', u'altman')
('feed', array([8605]))
('Sentence :', u'altman rake years regatta memotec pierre <unk> nonexecutive as will <eos> ssangyong director nahb group the cluett rubens snack-food fromstein calloway and memotec a board years regatta publishing fields rake group group rake cluett ssangyong pierre calloway memotec gitano gold rubens as as director sim is publishing gitano punts join <unk> and a old punts years memotec a rake is guterman cluett ssangyong will berlitz nahb <eos> of group join <unk> board join and pierre consolidated board cluett dutch gold as ipo ssangyong guterman a kia will dutch and director centrust consolidated rudolph guterman guterman cluett years n.v. old board rubens ')
================================================================================
('Loss :', 496.78199882507323)
('Batch_input :', array([4115, 5, 14, 45, 55, 3, 72, 195, 1244, 220, 2,
0, 3150, 7426, 1, 13, 4052, 1, 496, 14, 6885, 0,
1, 22, 113, 2652, 8068, 5, 14, 2474, 5250, 10, 464,
52, 3004, 466, 1244, 15, 2, 1, 80, 0, 167, 4,
35, 2645, 1, 65, 10, 558, 6092, 3574, 1898, 666, 1,
7, 27, 1, 4241, 6036, 7, 3, 2, 366], dtype=int32))
('Batch_labels :', array([[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
...,
[ 0., 0., 1., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.]], dtype=float32))
Average loss at step 100: 4.967820 learning rate: 1.000000
('Label: ', array([[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
...,
[ 0., 0., 1., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.],
[ 0., 0., 0., ..., 0., 0., 0.]], dtype=float32))
('Predicted:', array([[ 4.41551352e+00, 9.98007679e+00, 1.75690575e+01, ...,
6.83443546e+00, -2.30797195e+00, 1.73750782e+00],
[ 1.26826172e+01, 5.96618652e-03, 1.18247871e+01, ...,
-3.70885038e+00, -8.55356884e+00, -9.16959190e+00],
[ 1.44652233e+01, 5.12977028e+00, 9.42045784e+00, ...,
1.39444172e+00, 1.95213389e+00, -4.00810099e+00],
...,
[ 2.93052626e+00, 9.41266441e+00, 1.79130135e+01, ...,
4.24245834e+00, -1.46551771e+01, -3.35697136e+01],
[ 2.48945675e+01, 2.32091904e+01, 2.47276134e+01, ...,
-6.39845896e+00, -2.66628218e+00, -4.59843445e+00],
[ 1.34414902e+01, 4.80197811e+00, 1.89214745e+01, ...,
-5.91268682e+00, -8.80736637e+00, -6.49542713e+00]], dtype=float32))
================================================================================
[[ 0. 0. 0. ..., 0. 0. 0.]]
3619
('f', u'officially')
('as', u'officially')
('feed', array([3619]))
('Sentence :', u'officially <unk> to <eos> filters ago cigarettes is that cigarette stopped to <eos> researchers <unk> to <eos> filters ago cigarettes asbestos the filters ago cigarettes asbestos the filters ago cigarettes is that cigarette up the <eos> researchers to <eos> researchers <unk> to <eos> filters ago cigarettes asbestos the filters ago cigarettes asbestos <eos> filters ago cigarettes asbestos the filters ago cigarettes is that cigarette up the <eos> researchers <unk> to <eos> researchers <unk> to <eos> filters ago cigarettes asbestos of percentage years the the the <eos> researchers <unk> to <eos> filters ago cigarettes asbestos the filters ago cigarettes asbestos the filters ')
================================================================================
The initial loss is 0.92 which predicts the text as given.The next loss is around 4.57 at 100 step. But as the number of step increases loss increases which is anomaly (right?).
And also the next predicted word in output 'among' repeats at step 500.
Is there any error in training?
This is new output I get : pastebin link.
I am not 100% sure about the problem in your code but I noticed that you start the learning rate at 1.
learning_rate = tf.train.exponential_decay(1.0, global_step, 5000, 0.1, staircase=True)
Try to pick a lower initial value.
High learning rates causes the model weights to take long leaps so it may miss the minimum and it may even reach a point where the loss is higher (which may be your case). it is like super-jumping over a valley from one side to the other instead of going down into it.
Reference for the image: http://cs231n.github.io/neural-networks-3/
Lowering the learning rate from 1e-2 to 1e-4 solved a similar problem in a different model. Your model may work at a different learning rate.
Related
Relatively new to deeplearning, currently trying to implement a basic model with a custom loss function. The custom loss function is the main part of the code, compared to other parameters of the model.
1.I've attached the kind of loss functions used.
Loss function used in the code
Similar loss function
2.The loss function needs to iterate through the train and predict values and the loss is calculated correspondingly for negative and positive error. After dealing with the tensor related errors, like, "iterating through tensors" this current loss function is works without raising input related problems.
3.The loss function works as standalone.
do suggest any changes to the loss function, and any possible solutions to the current problem.
Ive attached the code for reference.
regarding the loss function, i'm not quite familiar with tf.scan, tf.map, keras.backend and such related functions yet which were suggested in many answers when dealing with loss function errors, but since the functions seems to be versatile and able to take data without an issue now, solution to the "current gradients problem" would be highly preferred.
Ive attached the code
`
def custom_loss_tensor(y_train,y_pred):
cs = 10.0
ch=1.0
loss = 0
y_train_t = tf.convert_to_tensor(y_train)
y_pred_t = tf.convert_to_tensor(y_pred)
num_train = y_train_t.numpy()
num_pred = y_pred_t.numpy()
l=len(num_train)
for i in range(l):
err = num_pred[i]-num_train[i]
if err < 0:
loss = loss + (10*abs(err))
else:
loss = loss + (1*abs(err))
return loss
model = Sequential()
model.add(Dense(43, kernel_initializer = 'normal', activation = 'relu'))
model.add(Dense(64, activation = 'relu'))
model.add(Dense(1))
model.compile(
loss = custom_loss_tensor,
optimizer = 'RMSprop',
metrics = keras.metrics.MeanAbsoluteError(),
run_eagerly=True)
`training = model.fit(
x_train, y_train,
batch_size=128,
epochs = 10,
verbose = 1
ValueError Traceback (most recent call last)
<ipython-input-60-dac93a08cc41> in <module>
3 batch_size=128,
4 epochs = 1,
----> 5 verbose = 1
6 )
E:\Anaconda\lib\site-packages\keras\utils\traceback_utils.py in error_handler(*args, **kwargs)
68 # To get the full stack trace, call:
69 # `tf.debugging.disable_traceback_filtering()`
---> 70 raise e.with_traceback(filtered_tb) from None
71 finally:
72 del filtered_tb
E:\Anaconda\lib\site-packages\keras\optimizers\optimizer_v2\utils.py in filter_empty_gradients(grads_and_vars)
76 variable = ([v.name for _, v in grads_and_vars],)
77 raise ValueError(
---> 78 f"No gradients provided for any variable: {variable}. "
79 f"Provided `grads_and_vars` is {grads_and_vars}."
80 )
ValueError: No gradients provided for any variable: (['dense_21/kernel:0', 'dense_21/bias:0', 'dense_22/kernel:0', 'dense_22/bias:0', 'dense_23/kernel:0', 'dense_23/bias:0'],). Provided `grads_and_vars` is ((None, <tf.Variable 'dense_21/kernel:0' shape=(43, 43) dtype=float32, numpy=
array([[-0.03749189, 0.17271727, -0.24716692, ..., -0.2605915 ,
-0.16543186, 0.18584403],
[-0.07696107, -0.02638303, -0.07502724, ..., 0.00748128,
0.02917111, -0.00045595],
[-0.15296972, -0.05007204, -0.09662418, ..., -0.10381483,
0.1687066 , 0.04201859],
...,
[-0.2555289 , 0.24466953, 0.19306567, ..., 0.09167928,
0.2091296 , 0.01142609],
[-0.03024916, -0.17035547, -0.10503584, ..., 0.22953227,
-0.06455661, -0.13004614],
[-0.16971609, -0.04739657, 0.23525235, ..., -0.06847623,
0.20173371, 0.12631011]], dtype=float32)>), (None, <tf.Variable 'dense_21/bias:0' shape=(43,) dtype=float32, numpy=
array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)>), (None, <tf.Variable 'dense_22/kernel:0' shape=(43, 64) dtype=float32, numpy=
array([[-0.01001556, 0.18438679, 0.0746735 , ..., -0.20779023,
-0.175497 , 0.10676223],
[-0.147724 , -0.05647631, 0.20822546, ..., -0.0742207 ,
0.13220128, 0.18023628],
[ 0.08255652, -0.15501451, -0.13425983, ..., -0.12613183,
-0.10449411, -0.09487195],
...,
[ 0.18172857, 0.08704039, 0.02496117, ..., -0.08698638,
0.00402144, 0.12817398],
[ 0.00345939, 0.07263863, 0.15878046, ..., -0.01657443,
-0.12882826, -0.15950364],
[ 0.06317642, -0.13567862, 0.03992519, ..., -0.11793269,
0.22939149, 0.17594114]], dtype=float32)>), (None, <tf.Variable 'dense_22/bias:0' shape=(64,) dtype=float32, numpy=
array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)>), (None, <tf.Variable 'dense_23/kernel:0' shape=(64, 1) dtype=float32, numpy=
array([[-0.2474489 ],
[ 0.27486765],
[-0.27261525],
[-0.17509465],
[ 0.09408 ],
[-0.02667353],
[-0.0183523 ],
[ 0.26157188],
[-0.17779922],
[ 0.20776463],
[ 0.05283326],
[ 0.30198514],
[-0.04326349],
[-0.21002822],
[-0.14250202],
[ 0.19137284],
[ 0.00679907],
[ 0.1577428 ],
[-0.2694474 ],
[-0.11011858],
[ 0.27931225],
[-0.23548083],
[-0.16819511],
[-0.01075685],
[ 0.21107608],
[ 0.22087872],
[ 0.11126944],
[ 0.04594085],
[ 0.1345087 ],
[ 0.14656761],
[-0.28515455],
[ 0.14429107],
[ 0.14043242],
[-0.09573163],
[ 0.19628167],
[ 0.1347841 ],
[-0.22662674],
[-0.25981647],
[ 0.00762352],
[-0.20713952],
[ 0.17875996],
[ 0.27148038],
[-0.0861142 ],
[-0.17500569],
[ 0.28790957],
[-0.02680674],
[ 0.14458871],
[-0.09571315],
[ 0.2938726 ],
[-0.1645372 ],
[ 0.21122003],
[-0.1245351 ],
[ 0.02794001],
[-0.1927064 ],
[-0.00268784],
[ 0.18155274],
[-0.07574154],
[-0.2926125 ],
[-0.05449736],
[ 0.16933608],
[-0.03486991],
[-0.09897752],
[-0.00102338],
[-0.09188385]], dtype=float32)>), (None, <tf.Variable 'dense_23/bias:0' shape=(1,) dtype=float32, numpy=array([0.], dtype=float32)>)).
You don't need to convert your tensors to numpy
def custom_loss_tensor(y_train,y_pred):
cs = 10.0
ch=1.0
loss = 0
for i in range(len(y_train)):
err = float(y_pred[i])-float(y_train[i])
if err < 0:
loss = loss + (10*abs(err))
else:
loss = loss + (1*abs(err))
return float(loss)
I have data predictor in 2D array like this below
array([[ 0, 0, 0, ..., 10, 6, 1],
[ 0, 0, 0, ..., 12, 6, 1],
[ 0, 0, 0, ..., 8, 6, 1],
...,
[ 0, 0, 0, ..., 54, 30, 60],
[ 0, 0, 0, ..., 1472, 5, 348],
[ 0, 0, 0, ..., 58, 45, 60]])
and label data with 3D shape like this below
[array([[0., 0., 1., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
...,
[0., 1., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.]], dtype=float32),
array([[0., 0., 1., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
...,
...
...,
[0., 1., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.]], dtype=float32),
...]
I would build the model with TensorFlow
model = Sequential()
model.add(Embedding(total_words, 10, input_length=max_sequence_len))
model.add(LSTM(100))
model.add(Dropout(0.1))
model.add(Dense(total_words_label, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam')
But, I get error in latest Dense
ValueError: Shapes (None, 10, 4991) and (None, 4991) are incompatible
It looks like from latest Dense shape because I have 3D label. How the way I can adjust it? I'm learning about TensorFlow.
And, how to convert the output to the 3D label when I have predict it?
--- update (adding the process) --
the example data
input output/label
Hartford Avenue, Bellingham, MA, 2019 {'address': 'Hartford Avenue', 'city': 'Bellingham', 'state': 'MA','zip': '2019'}
Oak Street, Brockton, MA, 2301 {'address': 'Oak Street', 'city': 'Brockton', 'state': 'MA', 'zip': '2301'}
the input is as predictor and the output/label is as label
The process of creating label
tokenizer_label = Tokenizer()
l = list(all['input'])
tokenizer_label.fit_on_texts(l)
total_words_label = len(tokenizer_label.word_index) + 1
labels = []
for i in range(len(l)):
labels.append(keras_utils.to_categorical(tokenizer_label.texts_to_sequences(l[i].split()), num_classes=total_words_label))
Please explain what tokenizer.texts_to_matrix does and what the result is?
from tensorflow.keras.preprocessing.text import Tokenizer
tokenizer = Tokenizer(oov_token="<OOV>")
sentences = [text]
print(sentences)
tokenizer.fit_on_texts(sentences)
word_index = tokenizer.word_index
sequences = tokenizer.texts_to_sequences(sentences)
matrix = tokenizer.texts_to_matrix(sentences)
print(word_index)
print(sequences)
print(matrix)
---
['The fool doth think he is wise, but the wise man knows himself to be a fool.']
# word_index
{'<OOV>': 1, 'the': 2, 'fool': 3, 'wise': 4, 'doth': 5, 'think': 6, 'he': 7, 'is': 8, 'but': 9, 'man': 10, 'knows': 11, 'himself': 12, 'to': 13, 'be': 14, 'a': 15}
# sequences
[[2, 3, 5, 6, 7, 8, 4, 9, 2, 4, 10, 11, 12, 13, 14, 15, 3]]
# matrix
[[0. 0. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]]
In the binary mode (default mode), it indicates which words from learnt vocabulary are in the input texts. You have trained your tokenizer on
['The fool doth think he is wise, but the wise man knows himself to be a fool.']
So when you convert the same text to a matrix, it will have all words (indicated by 1) except OOV - because all words are known - hence position at 1 of result vector is 0 (see word_index, and 0 is always 0 since words are enumerated starting from 1
Some examples
tokenizer.texts_to_matrix(['foo'])
# only OOV in this one text
array([[0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0.]])
tokenizer.texts_to_matrix(['he he'])
# known word, twice (does not matter how often)
array([[0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.,
0.]])
tokenizer.texts_to_matrix(['the fool'])
array([[0., 0., 1., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0.]])
Other mods
Other mods are more clear
count - How many times a word from vocabulary was in the text
tokenizer.texts_to_matrix(['He, he the fool'], mode="count")
array([[0., 0., 1., 1., 0., 0., 0., 2., 0., 0., 0., 0., 0., 0., 0., 0.,
0.]])
freq - count with sum normalized to 1.0
tokenizer.texts_to_matrix(['he he the fool'], mode="freq")
array([[0. , 0. , 0.25, 0.25, 0. , 0. , 0. , 0.5 , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. , 0. ]])
tfidf
tokenizer.texts_to_matrix(['he he the fool'], mode="tfidf")
array([[0. , 0. , 0.84729786, 0.84729786, 0. ,
0. , 0. , 1.43459998, 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. ]])
I created a VGG16 model using Keras application (TensorFlow backend). Then I wanted to change part of those weights and then test the accuracy of this modified model. To be direct and intuitive, I changed ALL the weights and biases in ALL layers to 0 like this:
model = VGG16(weights='imagenet', include_top=True)
# here is the test data and label containing 10 pictures I created.
data = np.load('./10_random_samples_array.npz')
data, label = data["X"], data["Y"]
# Modify the weights to zero
for z in [1, 2, 4, 5, 7, 8, 9, 11, 12, 13, 15, 16, 17]: # Conv layers
weight_bias = model.layers[z].get_weights()
shape_weight = np.shape(weight_bias[0])
shape_bias = np.shape(weight_bias[1])
weight_bias[0] = np.zeros(shape=(shape_weight[0],shape_weight[1],shape_weight[2],shape_weight[3]))
weight_bias[1] = np.zeros(shape=(shape_bias[0],))
model.layers[z].set_weights(weight_bias)
for z in [20,21,22]: # FC layers
weight_bias = model.layers[z].get_weights()
shape_weight = np.shape(weight_bias[0])
print(z, shape_weight)
shape_bias = np.shape(weight_bias[1])
weight_bias[0] = np.zeros(shape=(shape_weight[0],shape_weight[1],))
weight_bias[1] = np.zeros(shape=(shape_bias[0],))
model.layers[z].set_weights(weight_bias)
model.compile(loss='binary_crossentropy',
optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),
metrics=['accuracy'])
# To check if the weights have been modified.
print(model.layers[1].get_weights())
loss, acc = model.evaluate(data, label, verbose=1)
print(acc)
Then I got result like this:
[array([[[[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.]],
...(All zero, I omit them)
[[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.]]]], dtype=float32),
array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)]
10/10 [==============================] - 2s 196ms/step
0.9989999532699585
Firstly, You can tell that all the weights and biases have already been changed to 0 but the accuracy still stays very high. That is unreasonable.(The original result returned by model.evaluate() is 0.9993000030517578)
Secondly, I used only 10 pictures as my test dataset. The result must be a decimal with only one digit after the point. But I got 0.9989999532699585.
I also tried to modify all weights only in Conv1-1 to zero and the result is also 0.9989999532699585. It seems that it is the minimum result. Is there something wrong with my model? Or the weights cannot be modified in this way? Or model.evaluate() doesn't work as I suppose?
I used tf.nn.top_k()function from tensorflow to use the model's softmax probabilities to visualize the certainty of its predictions with 5 new images and with k=5. I have an output as follows which I am not sure how to exactly interpret. Could anyone explain the output please.
TopKV2(values=array([[ 1., 0., 0., 0., 0.],
[ 1., 0., 0., 0., 0.],
[ 1., 0., 0., 0., 0.],
[ 1., 0., 0., 0., 0.],
[ 1., 0., 0., 0., 0.]], dtype=float32), indices=array([[13, 0, 1, 2, 3],
[13, 0, 1, 2, 3],
[13, 0, 1, 2, 3],
[26, 0, 1, 2, 3],
[13, 0, 1, 2, 3]], dtype=int32))
From the documentation, it returns two tensors: the first with the top K value and the second with the indices of these values in the original tensor.
So for your data what I see is that the original tensor is always one-hot (i.e. has a single 1.0 entry per row and is 0 everywhere else).