I am using the tf.keras API and I want my Model to take input with shape (None,), None is batch_size.
The shape of keras.layers.Input() doesn't include batch_size, so I think it can't be used.
Is there a way to achieve my goal? I prefer a solution without tf.placeholder since it is deprecated
By the way, my model is a sentence embedding model, so I want the input is something like ['How are you.','Good morning.']
======================
Update:
Currently, I can create an input layer with layers.Input(dtype=tf.string,shape=1), but this need my input to be something like [['How are you.'],['Good morning.']]. I want my input to have only one dimension.
Have you tried tf.keras.layers.Input(dtype=tf.string, shape=())?
If you wanted to set a specific batch size, tf.keras.Input() does actually include a batch_size parameter. But the batch size is presumed to be None by default, so you shouldn't even need to change anything.
Now, it seems like what you actually want is to be able to provide samples (sentences) of variable length. Good news! The tf.keras.layers.Embedding layer allows you to do this, although you'll have to generate an encoding for your sentences first. The Tensorflow website has a good tutorial on the process.
Related
I would like to change the input and output size of a convolutional model of tensorflow, which I am importing from the tensorflow hub.
Would I like to know what is the best way to do this? If I could convert the model to kaeras format I think it would be easier, but I'm not succeeding either.
This is the model https://tfhub.dev/intel/midas/v2_1_small/1
The format of the input is determined by the publisher of the model. Some models could be flexible on the dimensions of the input and some require input with very specific dimensions. In that case, the best way would be to resize the input as needed before feeding it to the model.
I am trying to code a simple Neural machine translation using tensorflow. But I am a little stuck regarding the understanding of the embedding on tensorflow :
I do not understand the difference between tf.contrib.layers.embed_sequence(inputs, vocab_size=target_vocab_size,embed_dim=decoding_embedding_size)
and
dec_embeddings = tf.Variable(tf.random_uniform([target_vocab_size, decoding_embedding_size]))
dec_embed_input = tf.nn.embedding_lookup(dec_embeddings, dec_input)
In which case should I use one to another ?
The second thing I do not understand is about tf.contrib.seq2seq.TrainingHelper and tf.contrib.seq2seq.GreedyEmbeddingHelper. I know that in the case of translation, we use mainly TrainingHelper for the training step (use the previous target to predict the next target) and GreedyEmbeddingHelper for the inference step (use the previous timestep to predict the next target).
But I do not understand how does it work. In particular the different parameters used. For example why do we need a sequence length in the case of TrainingHelper (why do we not used an EOS)? Why both of them do not use the embedding_lookup or embedding_sequence as input ?
I suppose that you're coming from this seq2seq tutorial. Even though this question is starting to get old, I'll try to answer for the people passing by like me:
For the first question, I looked at the source code behind tf.contrib.layers.embed_sequence, and it is actually using tf.nn.embedding_lookup. So it just wraps it, and creates the embedding matrix (tf.Variable(tf.random_uniform([target_vocab_size, decoding_embedding_size]))) for you. Although this is convenient and less verbose, by using embed_sequence there doesn't seem to a direct way to access the embeddings. So if you want to, you have to query for the internal variable used as the embedding matrix by using the same name space. I have to admit that the code in the tutorial above is confusing. I even suspect he's using different embeddings in the encoder and the decoder.
For the second question:
I guess it is equivalent to use a sequence length or an embedding.
The TrainingHelper doesn't need the embedding_lookup as it only forwards the inputs to the decoder, GreedyEmbeddingHelper does take as a first input the embedding_lookup as mentioned in the documentation.
If I understand you correctly, the first question is about the differences between tf.contrib.layers.embed_sequence and tf.nn.embedding_lookup.
According to the official docs (https://www.tensorflow.org/api_docs/python/tf/contrib/layers/embed_sequence),
Typical use case would be reusing embeddings between an encoder and decoder.
I think tf.contrib.layers.embed_sequence is designed for seq2seq models.
I found the following post:
https://github.com/tensorflow/tensorflow/issues/17417
where #ispirmustafa mentioned:
embedding_lookup doesn't support invalid ids.
Also, in another post: tf.contrib.layers.embed_sequence() is for what?
#user1930402 said:
When building a neural network model that has multiple gates that take features as input, by using tensorflow.contrib.layers.embed_sequence, you can reduce the number of parameters in your network while preserving depth. For example, it eliminates the need for each gates of the LSTM to perform its own linear projection of features.
It allows for arbitrary input shapes, which helps the implementation be simple and flexible.
For the second question, sorry that I didn't use TrainingHelper and can't answer your question.
I have some questions on the use of the embedding columns implemented in tensorflow in tf.contrib.layers.embedding_column.
I'm training a binary classifier for a two player game (tennis, in this case). My features that I pass to the DNNClassifier as feature_columns look as follows:
deep_columns = [
tf.contrib.layers.embedding_column(player1, dimension=9),
tf.contrib.layers.embedding_column(player2, dimension=9),
tf.contrib.layers.embedding_column(court, dimension=1),
tf.contrib.layers.embedding_column(surface, dimension=1),
p1Rank, p2Rank]
What I'm wondering about is this: am I now learning two different embeddings for the same set of players? And if so, is there a way to use one embedding layer for both players? Or is there nothing wrong with doing it the way I'm currently doing it?
A second question regarding the embedding_column: the docs mention this, as possible arguments for embedding_column:
ckpt_to_load_from: (Optional). String representing checkpoint name/pattern to restore the column weights. Required if tensor_name_in_ckpt is not None.
tensor_name_in_ckpt: (Optional). Name of the Tensor in the provided checkpoint from which to restore the column weights. Required if ckpt_to_load_from is not None.
Does this imply that if none of these is provided, the embedding layers are initialized randomly again when restoring my model from a checkpoint?
And then one final question: The two embedding columns for court and surface have a dimension of 1, as they only have very few options. Is this a bad use of an embedding column? Or is it okay to use it like that?
Thanks in advance!
I am looking for a proper or best way to get variable importance in a Neural Network created with Keras. The way I currently do it is I just take the weights (not the biases) of the variables in the first layer with the assumption that more important variables will have higher weights in the first layer. Is there another/better way of doing it?
Since everything will be mixed up along the network, the first layer alone can't tell you about the importance of each variable. The following layers can also increase or decrease their importance, and even make one variable affect the importance of another variable. Every single neuron in the first layer itself will give each variable a different importance too, so it's not something that straightforward.
I suggest you do model.predict(inputs) using inputs containing arrays of zeros, making only the variable you want to study be 1 in the input.
That way, you see the result for each variable alone. Even though, this will still not help you with the cases where one variable increases the importance of another variable.
*Edited to include relevant code to implement permutation importance.
I answered a similar question at Feature Importance Chart in neural network using Keras in Python. It does implement what Teque5 mentioned above, namely shuffling the variable among your sample or permutation importance using the ELI5 package.
from keras.wrappers.scikit_learn import KerasClassifier, KerasRegressor
import eli5
from eli5.sklearn import PermutationImportance
def base_model():
model = Sequential()
...
return model
X = ...
y = ...
my_model = KerasRegressor(build_fn=basemodel, **sk_params)
my_model.fit(X,y)
perm = PermutationImportance(my_model, random_state=1).fit(X,y)
eli5.show_weights(perm, feature_names = X.columns.tolist())
It is not that simple. For example, in later stages the variable could be reduced to 0.
I'd have a look at LIME (Local Interpretable Model-Agnostic Explanations). The basic idea is to set some inputs to zero, pass it through the model and see if the result is similar. If yes, then that variable might not be that important. But there is more about it and if you want to know it, then you should read the paper.
See marcotcr/lime on GitHub.
This is a relatively old post with relatively old answers, so I would like to offer another suggestion of using SHAP to determine feature importance for your Keras models. SHAP also allows you to process Keras models using layers requiring 3d input like LSTM and GRU while eli5 cannot.
To avoid double-posting, I would like to offer my answer to a similar question on Stackoverflow on using SHAP.
I'm trying to use TensorFlow to train output servo commands given an input image.
I plan on using a file as #mrry suggested in this question, with the images like so:
../some/path/some_img.JPG *some_label*
My question is, what are the label formats I can provide to TensorFlow and what structures are suggested?
My data is basically n servo commands from 0-10 seconds. A vector would work great:
[0,2,4,3]
or similarly:
[0,.25,.4,.3]
I couldn't find much about labels in the docs. Can anyone shed any light on TensorFlow labels?
And a very related question is what is the best way to structure these for TensorFlow to properly learn from them?
In Tensorflow Labels are just generic tensor. You can use any kind of tensor to store your labels. In your case a 1-D tensor with shape (4,) seems to be desired.
Labels do only differ from the rest of the data by its use in the computational graph. (Usually) labels should only be used inside the loss function while you propagate the other data through the whole network. For your problem a 4-d regression function should work.
Also, look at my newest comment to the (old) question. Using the slice_input_producer seems to be preferable in your case.