Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
Can we use Bahdanau attention for multivariate time-series prediction problem? Using the Bahdanau implementation from here, I have come up with following code for time series prediction.
from tensorflow.keras.layers import Input, LSTM, Concatenate, Flatten
from attention_keras import AttentionLayer
from tensorflow.keras import Model
num_inputs = 5
seq_length = 10
inputs = Input(shape=(seq_length, num_inputs), name='inputs')
lstm_out = LSTM(64, return_sequences=True)(inputs)
lstm_out = LSTM(64, return_sequences=True)(lstm_out)
# Attention layer
attn_layer = AttentionLayer(name='attention_layer')
attn_out, attn_states = attn_layer([lstm_out, lstm_out])
# Concat attention input and LSTM output, in original code it was decoder LSTM
concat_out = Concatenate(axis=-1, name='concat_layer')([lstm_out, attn_out])
flat_out = Flatten()(concat_out)
# Dense layer
dense_out = Dense(seq_length, activation='relu')(flat_out)
predictions= dense_time(1)(dense_out)
# Full model
full_model = Model(inputs=inputs, outputs=predictions)
full_model.compile(optimizer='adam', loss='mse')
For my data, the model does perform better than vanilla LSTM without attention, but I am not sure if this implementation make sense or not?
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed last year.
Improve this question
i am using du functional api from keras and would like to add a dropout to my multi layer perceptron.
do i have to put the dropout before or after the layer and do i have to connect the next layer to the dropout or to the previous layer?
hidden_Layer_2 = Dense(152,activation='relu')(hidden_Layer_1)
dropout_2 = Dropout(0.4)(hidden_Layer_2)
hidden_Layer_3 = Dense(152, activation='relu')(hidden_Layer_2)
or
hidden_Layer_2 = Dense(152,activation='relu')(hidden_Layer_1)
dropout_2 = Dropout(0.4)(hidden_Layer_2)
hidden_Layer_3 = Dense(152, activation='relu')(dropout_2 )
The second option is the right one. You always need to connect the layers in the order you want to use them.
hidden_Layer_2 = Dense(152,activation='relu')(hidden_Layer_1)
dropout_2 = Dropout(0.4)(hidden_Layer_2)
hidden_Layer_3 = Dense(152, activation='relu')(dropout_2 )
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 1 year ago.
Improve this question
I want my Neural network output to be either 0 or 1, and not probabilities between 0 and 1.
for the same I have designed step function for output layer, I want my output layer to just roundoff the output of previous(softmax) layer i.e. converting probabilities into 0 and 1.
My customised function is not giving the expected results .
Kindly help.
My code is :
from keras.layers.core import Activation
from keras.models import Sequential
from keras import backend as K
# Custom activation function
from keras.layers import Activation
from keras import backend as K
from keras.utils.generic_utils import get_custom_objects
#tf.custom_gradient
def custom_activation(x):
print("tensor ",x)
ones = tf.ones(tf.shape(x), dtype=x.dtype.base_dtype)
zeros = tf.zeros(tf.shape(x), dtype=x.dtype.base_dtype)
def grad(dy):
return dy
print(" INSIDE ACTOVATION FUNCTION ")
return keras.backend.switch(x > .5, ones, zeros), grad
model = keras.models.Sequential()
model.add(keras.layers.Dense(32,input_dim=a,activation='relu'))
model.add(keras.layers.Dropout(0.2))
model.add(keras.layers.Dense(64,activation="relu"))
model.add(keras.layers.Dropout(0.2))
model.add(keras.layers.Dense(2,activation='softmax'))
model.add(Activation(custom_activation, name='custom_activation'))#output layer
### Compile the model
model.compile(loss="binary_crossentropy",optimizer='adam',metrics=["accuracy"])
First of all, note that obtaining the class probabilities is always yielding more information than a pure 0-1 classification, and thus your model will almost always train better and faster.
That being said, and considering that you do have an underlying reason to limit your NN, a hard decision like the one you want to implement as activaction function is know as step function or heaviside function. The main problem with this function is that, by default, the function is non-differentiable (there is an infinite slope in the threshold, 0.5 in your case). To address this you have two options:
Create a custom "approximative" gradient that is differentiable. This SO answer covers it well.
Use tf.cond(), which, relying on TF's AutoGrad, will only execute one branch of the graph at runtime, and omit the unused branch.
class MyHeavisideActivation(tf.keras.layers.Layer):
def __init__(self, num_outputs, threshold=.5, **kwargs):
super(MyHeavisideActivation, self).__init__(**kwargs)
self.num_outputs = num_outputs
self.threshold = threshold
def build(self, input_shape):
pass
def call(self, inputs):
return tf.cond(inputs > self.threshold,
lambda: tf.add(tf.multiply(inputs,0), 1), # set to 1
lambda: tf.multiply(inputs, 0)) # set to 0
#> ...same as above
model.add(keras.layers.Dense(2,activation='softmax'))
model.add(MyHeavisideActivation(2, name='custom_activation'))#output layer
Closed. This question is not reproducible or was caused by typos. It is not currently accepting answers.
This question was caused by a typo or a problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers.
Closed 2 years ago.
Improve this question
import tensorflow as tf
from tensorflow.keras.datasets import mnist
from tensorflow.keras import models
from tensorflow.keras import layers
from tensorflow.keras.utils import to_categorical
network=models.Sequential() # this initializes a sequential model that we will call network
network.add(layers.Dense(10, activation = 'relu') # this adds a dense hidden layer
network.add(layers.Dense(8, activation = 'softmax')) # this is the output layer
I am trying to create a 2 layer neural network model in tensorflow and am getting this error:
File "<ipython-input-6-0dde2ff676f8>", line 7
network.add(layers.Dense(8, activation = 'softmax')) # this is the output layer
^
SyntaxError: invalid syntax
May I know why I'm getting this error for output layer but not for hidden layer? Thanks.
You have missed a closing bracket.
import tensorflow as tf
from tensorflow.keras.datasets import mnist
from tensorflow.keras import models
from tensorflow.keras import layers
from tensorflow.keras.utils import to_categorical
network=models.Sequential() # this initializes a sequential model that we will call network
network.add(layers.Dense(10, activation = 'relu')) # this adds a dense hidden layer
network.add(layers.Dense(8, activation = 'softmax')) # this is the output layer
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
I followed a tutorial about detecting objects using deep learning here: https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html
At some point, after training up to 4082 steps, i stopped the training using CTRL+C.
Now i have bunch of files under my training directory, which looks like this:
list of files in the training directory
The question is, how do i proceed now? what to do next? the tutorial doesn't teach you how to use the training data, how to even test it if its recognizing correctly.
Thanks in advance.
The files you obtained are checkpoints. What you want to do now is to restore your model from the checkpoints. Indications from CV tricks:
with tf.Session() as sess:
model = tf.train.import_meta_graph('my_test_model-1000.meta')
model.restore(sess, tf.train.latest_checkpoint('./'))
After, you can evaluate your model on your test set:
test_accuracy = tfe.metrics.Accuracy()
for (x, y) in test_dataset:
logits = model(x)
prediction = tf.argmax(logits, axis=1, output_type=tf.int32)
test_accuracy(prediction, y)
print("Test set accuracy: {:.3%}".format(test_accuracy.result()))
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I'm not exactly sure how to approach this question. The dataset has 8 attributes and one y-value. How would I train a linear regression model on 85% of the dataset?
You can also use train_test_split from sklearn as in sklearn example to split the data into training and testing sets e.g. if X is data with features and y is label then:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.15)
And for linear regression you can try using: linregress from scipy as in similar question:
Use ndf = df.sample(frac=0.85) to get a DataFrame with 85% of your total rows and then use this new DataFrame ndf to train your linear regression model.