Related
I'm trying to train LSTM model in Keras using data of variable timestep, for example, the data looks like:
<tf.RaggedTensor [[[0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0],
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]],
[[1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
[1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]],
[[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0]], ...,
[[0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]],
[[1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0],
[1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0],
[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0],
[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0],
[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]],
[[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]]]>
and its corresponding label:
<tf.RaggedTensor [[6, 6], [7, 7], [8], ..., [6], [11, 11, 11, 11, 11], [24, 24, 24, 24, 24]]>
Each input data have 13 features, so for each time step, the model receives a 1 x 13 vector. I wonder if it is possible to do so? I don't mind doing this on pytorch either.
I try to align them with no reshape layer.
However, my input for each time step in the LSTM layer is a vector of dimension 13. And each sample has variable-length of these vectors, which means the time step is not constant for each sample. Can you show me a code example of how to train such model? –
TurquoiseJ
First of all, the concept of windows length and time steps is they take the same amount of the input with a higher number of length and time.
We assume the input to extract features can be divide by multiple times of windows travels along with axis, please see the attached for idea.
[Codes]:
batched_features = tf.constant( [ [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ], [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ], ], shape=( 2, 1, 13 ) )
batched_labels = tf.constant( [[ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ], [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]], shape=( 2, 13 ) )
dataset = tf.data.Dataset.from_tensor_slices((batched_features, batched_labels))
dataset = dataset.batch(10)
batched_features = dataset
[Sample]:
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
bidirectional (Bidirectiona (None, 1, 64) 11776
l)
bidirectional_1 (Bidirectio (None, 64) 24832
nal)
dense (Dense) (None, 13) 845
=================================================================
Total params: 37,453
Trainable params: 37,453
Non-trainable params: 0
_________________________________________________________________
<BatchDataset element_spec=(TensorSpec(shape=(None, 1, 13), dtype=tf.int32, name=None), TensorSpec(shape=(None, 13), dtype=tf.int32, name=None))>
Epoch 1/100
2022-03-28 05:19:04.116345: I tensorflow/stream_executor/cuda/cuda_dnn.cc:368] Loaded cuDNN version 8100
1/1 [==============================] - 8s 8s/step - loss: 0.0000e+00 - accuracy: 1.0000 - val_loss: 0.0000e+00 - val_accuracy: 1.0000
Epoch 2/100
1/1 [==============================] - 0s 38ms/step - loss: 0.0000e+00 - accuracy: 1.0000 - val_loss: 0.0000e+00 - val_accuracy: 1.0000
Assume each windows consume about 13 level of the input :
batched_features = tf.constant( [ [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ], [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ], ], shape=( 2, 1, 13 ) )
batched_labels = tf.constant( [[ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ], [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]], shape=( 2, 13 ) )
Adding more windows is easy by
batched_features = tf.constant( [ [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ], [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ], [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ], ], shape=( 3, 1, 13 ) )
batched_labels = tf.constant( [[ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ], [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ], [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]], shape=( 3, 13 ) )
dataset = tf.data.Dataset.from_tensor_slices((batched_features, batched_labels))
dataset = dataset.batch(10)
batched_features = dataset
At least you tell me what is the purpose they can use reverse windows to have certain results. ( Apmplitues frequency )
The results will look like these for each windows :
[ Output ] : 2 and 3 Windows
# Sequence types with timestep #1:
# <BatchDataset element_spec=(TensorSpec(shape=(None, 1, 13), dtype=tf.int32, name=None), TensorSpec(shape=(None, 13), dtype=tf.int32, name=None))>
# Sequence types with timestep #2:
# <BatchDataset element_spec=(TensorSpec(shape=(None, 1, 13), dtype=tf.int32, name=None), TensorSpec(shape=(None, 13), dtype=tf.int32, name=None))>
[ Result ]:
I am plotting multiple instances of data that use the same label. There are techniques to remove duplicates from the legend, shown here.
However, I additionally want to prevent different colors being used for the same label.
If I call
import matplotlib.pyplot as plt
plt.plot([1,2,3], [1.0, 3.5, 2.5], label='a')
plt.plot([1,2,3], [1.5, 4.0, 3.0], label='a')
plt.plot([1,2,3], [2.0, 3.0, 3.1], label='b')
plt.plot([1,2,3], [1.5, 2.5, 1.0], label='b')
ax = plt.gca()
handles, labels = ax.get_legend_handles_labels()
unique = [(h, l) for i, (h, l) in enumerate(zip(handles, labels)) if l not in labels[:i]]
ax.legend(*zip(*unique))
I am hoping for a plot like:
However, I get this plot:
Is there a way I can get the former plot without having to explicitly set the colors, like so:
plt.plot([1,2,3], [1.0, 3.5, 2.5], label='a', color='C0')
plt.plot([1,2,3], [1.5, 4.0, 3.0], label='a', color='C0')
plt.plot([1,2,3], [2.0, 3.0, 3.1], label='b', color='C1')
plt.plot([1,2,3], [1.5, 2.5, 1.0], label='b', color='C1')
I think the most straight-forward would be to just remove the labels for all but one of the lines for each label:
plt.plot([1,2,3], [1.0, 3.5, 2.5], label='a', color='C0')
plt.plot([1,2,3], [1.5, 4.0, 3.0], color='C0')
plt.plot([1,2,3], [2.0, 3.0, 3.1], label='b', color='C1')
plt.plot([1,2,3], [1.5, 2.5, 1.0], color='C1')
plt.legend()
You could achieve the same by using an initial underscore for any labels you don't want to show:
plt.plot([1,2,3], [1.0, 3.5, 2.5], label='a', color='C0')
plt.plot([1,2,3], [1.5, 4.0, 3.0], label='_a', color='C0')
plt.plot([1,2,3], [2.0, 3.0, 3.1], label='b', color='C1')
plt.plot([1,2,3], [1.5, 2.5, 1.0], label='_b', color='C1')
plt.legend()
I find this sometimes useful for plotting in a loop (inserting '_' to the start of a label depending on the iteration in the loop, perhaps including a list of colors to set the color for each line - though this would be trivial in your example).
I can't think of a scenario where you actually need matplotlib to "know" that each line is part of the same label. These type of approaches have been fine in my experience.
I have a large number of .csv files with the following cell values:
"[[0.0, 4.0], .... , [240.0, 0.0], [248.0, 0.0]]"
The string contains a nested list and is a result of a histogram reducer with 32 bins for 8bit data and contains the lower bin value and the count.
For instance, the first element contains the lower bin value of the 1st bin (0.0) and the count (4.0). The last element contains the lower bin value of the 32nd bin (248.0) and count (0.0).
Since the lower bin values do not change and are known [0,8,16 ... 248], I would like to extract only the counts i.e.
[4, .... , 0 ]
In Python, this would be straight forward, however the amount of data is quite big and I have 3,422,250 of these histograms. Therefore I considered using Google BigQuery to get the job done.
When I load the cvs data in BigQuery, the histograms are stored as type STRING.
How can I get nested lists (arrays) that are stored as string in csv, in the ARRAY datatype in BigQuery? In the documentation, it says that nested arrays are not yet supported. Are there workarounds?
guidance on how to get the first element of multiple arrays is very welcome too!
p.s. I already tried to solve the problem upstream to no avail.
Example csv file
Not sure if it is exactly what you are asking, but hope below example (for BigQuery Standard SQL) will help you
#standardSQL
WITH `project.dataset.table` AS (
SELECT 1 id,'[[0.0, 4.0], [8.0, 0.0], [16.0, 0.0], [24.0, 0.0], [32.0, 0.0], [40.0, 0.0], [48.0, 0.0], [56.0, 0.0], [64.0, 1.0], [72.0, 1.0], [80.0, 4.0], [88.0, 0.0], [96.0, 0.0], [104.0, 0.0], [112.0, 0.0], [120.0, 0.0], [128.0, 0.0], [136.0, 0.0], [144.0, 0.0], [152.0, 0.0], [160.0, 0.0], [168.0, 0.0], [176.0, 0.0], [184.0, 0.0], [192.0, 0.0], [200.0, 0.0], [208.0, 0.0], [216.0, 0.0], [224.0, 0.0], [232.0, 0.0], [240.0, 0.0], [248.0, 0.0]]' histogram UNION ALL
SELECT 2, '[[0.0, 0.0], [8.0, 0.0], [16.0, 0.0], [24.0, 0.0], [32.0, 0.0], [40.0, 0.0], [48.0, 0.0], [56.0, 0.0], [64.0, 0.0], [72.0, 0.0], [80.0, 0.0], [88.0, 0.0], [96.0, 0.0], [104.0, 0.0], [112.0, 1.0], [120.0, 0.0], [128.0, 1.0], [136.0, 0.0], [144.0, 0.0], [152.0, 0.0], [160.0, 0.0], [168.0, 0.0], [176.0, 0.0], [184.0, 0.0], [192.0, 0.0], [200.0, 0.0], [208.0, 0.0], [216.0, 0.0], [224.0, 0.0], [232.0, 0.0], [240.0, 0.0], [248.0, 0.0]]'
)
SELECT id,
SPLIT(bin)[OFFSET(0)] value,
SPLIT(bin)[OFFSET(1)] frequency
FROM `project.dataset.table`, UNNEST(SPLIT(REGEXP_REPLACE(histogram, r'\[\[|]]|\s', ''), '],[')) bin
Note: this assumes When I load the cvs data in BigQuery, the histograms are stored as type STRING as
"[[0.0, 4.0], .... , [240.0, 0.0], [248.0, 0.0]]"
OR - if you want to keep rows intact and have histogram presented as string to be transformed into array - you can try below
#standardSQL
WITH `project.dataset.table` AS (
SELECT 1 id,'[[0.0, 4.0], [8.0, 0.0], [16.0, 0.0], [24.0, 0.0], [32.0, 0.0], [40.0, 0.0], [48.0, 0.0], [56.0, 0.0], [64.0, 1.0], [72.0, 1.0], [80.0, 4.0], [88.0, 0.0], [96.0, 0.0], [104.0, 0.0], [112.0, 0.0], [120.0, 0.0], [128.0, 0.0], [136.0, 0.0], [144.0, 0.0], [152.0, 0.0], [160.0, 0.0], [168.0, 0.0], [176.0, 0.0], [184.0, 0.0], [192.0, 0.0], [200.0, 0.0], [208.0, 0.0], [216.0, 0.0], [224.0, 0.0], [232.0, 0.0], [240.0, 0.0], [248.0, 0.0]]' histogram UNION ALL
SELECT 2, '[[0.0, 0.0], [8.0, 0.0], [16.0, 0.0], [24.0, 0.0], [32.0, 0.0], [40.0, 0.0], [48.0, 0.0], [56.0, 0.0], [64.0, 0.0], [72.0, 0.0], [80.0, 0.0], [88.0, 0.0], [96.0, 0.0], [104.0, 0.0], [112.0, 1.0], [120.0, 0.0], [128.0, 1.0], [136.0, 0.0], [144.0, 0.0], [152.0, 0.0], [160.0, 0.0], [168.0, 0.0], [176.0, 0.0], [184.0, 0.0], [192.0, 0.0], [200.0, 0.0], [208.0, 0.0], [216.0, 0.0], [224.0, 0.0], [232.0, 0.0], [240.0, 0.0], [248.0, 0.0]]'
)
SELECT id,
ARRAY(
SELECT AS STRUCT
SPLIT(bin)[OFFSET(0)] value,
SPLIT(bin)[OFFSET(1)] frequency
FROM UNNEST(SPLIT(REGEXP_REPLACE(histogram, r'\[\[|]]|\s', ''), '],[')) bin
) histogram_as_array
FROM `project.dataset.table`
I have a tensor 'a', I want to modify a element of it.
a = tf.convert_to_tensor([[1.0, 1.0, 1.0],
[1.0, 2.0, 1.0],
[1.0, 1.0, 1.0]], dtype=tf.float32)
And I can got the index of that element.
index = tf.where(a==2)
How to derive 'b' from 'a'?
b = tf.convert_to_tensor([[1.0, 1.0, 1.0],
[1.0, 0.0, 1.0],
[1.0, 1.0, 1.0]], dtype=tf.float32)
I know that I can't not modify a tensor from this post.
I solve it by using tf.sparse_to_dense()
import tensorflow as tf
a = tf.convert_to_tensor([[1.0, 1.0, 1.0],
[1.0, 2.0, 1.0],
[1.0, 1.0, 1.0]], dtype=tf.float32)
index = tf.where(a > 1)
zero = tf.sparse_to_dense(index, tf.shape(a, out_type=tf.int64), 0., 1.)
update = tf.sparse_to_dense(index, tf.shape(a, out_type=tf.int64), 0., 0.)
b = a * zero + update
with tf.Session() as sess:
print sess.run(b)
Need to plot a confusion matrix with this script. By running it an empty plot appears. Seems I am close to solution. Any hint?
from numpy import *
import matplotlib.pyplot as plt
from pylab import *
conf_arr = [[50.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [3.0, 26.0, 0.0, 0.0, 0.0, 1.0, 0.0], [0.0, 0.0, 10.0, 0.0, 0.0, 0.0, 0.0], [4.0, 1.0, 0.0, 5.0, 0.0, 0.0, 0.0], [3.0, 0.0, 1.0, 0.0, 6.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0, 0.0, 47.0, 0.0], [2.0, 0.0, 0.0, 0.0, 0.0, 0.0, 8.0]]
norm_conf = []
for i in conf_arr:
a = 0
tmp_arr = []
a = sum(i,0)
for j in i:
tmp_arr.append(float(j)/float(a))
norm_conf.append(tmp_arr)
plt.clf()
fig = plt.figure()
ax = fig.add_subplot(111)
res = ax.imshow(array(norm_conf), cmap=cm.jet, interpolation='nearest')
cb = fig.colorbar(res)
savefig("confmat.png", format="png")
Thanks, I have the plot. Now, the ticks in the x-axes are very small (the graph dimension is: 3 cm x 10 cm or so). How can I enlarge them in order to have a more proportioned graph, lets say 10cm x 10 cm plot? A possible reason is that I visualize the graph as a subplot? Was not able to find the suitable literature to adjust that.
You don't need to clear a current figure (plt.clf()) before adding a new one.
#plt.clf() # <<<<< here
fig = plt.figure()
ax = fig.add_subplot(111)