I would like to convert the ActorDistributionModel from a trained PPOClipAgent into a Tensorflow Lite model for deployment. How should I accomplish this?
I have tried following this tutorial (see section at bottom converting policy to TFLite), but the network outputs a single action (the policy) rather than the density function over actions that I desire.
I think perhaps something like this could work:
tf.compat.v2.saved_model.save(actor_net, saved_model_path, signature=?)
... if I knew how to set the signature parameter. That line of code executes without error when I omit the signature parameter, but I get the following error on load (I assume because the signature is not set up correctly):
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_path)
File "/home/ais/salesmentor.ai/MDPSolver/src/solver/ppo_budget.py", line 336, in train_eval
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_path)
File "/home/ais/.local/lib/python3.9/site-packages/tensorflow/lite/python/lite.py", line 1275, in from_saved_model
raise ValueError("Only support a single signature key.")
ValueError: Only support a single signature key.
This appears to work. I won't accept the answer until I have completed an end-to-end test, though.
def export_model(actor_net, observation_spec, saved_model_path):
predict_signature = {
'action_pred':
tf.function(func=lambda x: actor_net(x, None, None)[0].logits,
input_signature=(tf.TensorSpec(shape=observation_spec.shape),)
)
}
tf.saved_model.save(actor_net, saved_model_path, signatures=predict_signature)
# Convert to TensorFlow Lite model.
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_path,
signature_keys=["action_pred"])
converter.target_spec.supported_ops = [
tf.lite.OpsSet.TFLITE_BUILTINS, # enable TensorFlow Lite ops.
tf.lite.OpsSet.SELECT_TF_OPS # enable TensorFlow ops.
]
tflite_policy = converter.convert()
with open(os.path.join(saved_model_path, 'policy.tflite'), 'wb') as f:
f.write(tflite_policy)
The solution wraps the actor_net in a lambda because I was unable to figure out how to specify the signature with all three expected arguments. Through the lambda, I convert the function into using a single argument (a tensor). I expect to pass None to the other two arguments in my use case, so there is nothing lost in this approach.
I see you using CartPole as the model simulation, Agent DQN, and Model learning and Evaluation from links provided TF-Agent Checkpointer. For simple understanding, you need to understand about the distributions and your model limits ( less than 6 actions determining at a time ).
Discretes Distribution, answer the question to the points but the links is how they implement AgentDQN on TF- Agent.
temp = tf.random.normal([10], 1, 0.2, tf.float32), mean is one and the standard deviation is 0.2. Overall of result summation product is nearby one and its variance is 0.2, when they have 10 actions to determine the possibility of the result is the same action is 1 from 5 or 0.5. random normal
Coefficient is ladder steps or you understand as IF and ELSE conditions or SWITCH conditions such as at the gap of 0 to 5, 5 to 10, 10 to 15, and continue.
The matrixes product from the Matrix coefficients and randoms is selected 4 - 5 actions sorted by priority, significant and select the most effects in rows.
The ArgMax is 0 to 9 which is actions 0 - 9 that respond to the environment input co-variances.
Sample: To the points, random distributions and selective agents ( we call selective agent maybe the questioner has confused with NN DQN )
temp = tf.random.normal([10], 1, 0.2, tf.float32)
temp = np.asarray(temp) * np.asarray([ coefficient_0, coefficient_1, coefficient_2, coefficient_3, coefficient_4, coefficient_5, coefficient_6, coefficient_7, coefficient_8, coefficient_9 ])
temp = tf.nn.softmax(temp)
action = int(np.argmax(temp))
My idea is to extract the CLS token for all the text in the DB and save it in CSV or somewhere else. So when a new text comes in, instead of using the Cosine Similarity/JAccard/MAnhattan/Euclidean or other distances, I have to use some approximation like LSH, ANN (ANNOY, sklearn.neighbor) or the one given here faiss . How can that be done? I have my code as:
PyTorch:
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')
input_ids = torch.tensor(tokenizer.encode("Hello, I am a text")).unsqueeze(0) # Batch size 1
outputs = model(input_ids)
last_hidden_states = outputs[0] # The last hidden-state is the first element of the output tuple
Using Tensorflow:
import tensorflow as tf
from transformers import BertTokenizer, TFBertModel
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = TFBertModel.from_pretrained('bert-base-uncased')
input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute"))[None, :] # Batch size 1
outputs = model(input_ids)
last_hidden_states = outputs[0] # The last hidden-state is the first element of the output tuple
and I think can get the CLS token as: (Please correct if wrong)
last_hidden_states = outputs[0]
cls_embedding = last_hidden_states[0][0]
Please tell me if it's the right way to use and how can I use any of the LSH, ANNOT, faiss or something like that?
So for every text, there'll a 768 length vector and we can create a N(No of texts 10M)x768 matrix, how can I find the Index of top-5 data points (texts) which are most similar to the given image/embedding/data point?
I have a data a very simple one to test on my understanding about the usage of tf.padded_batch
text file is saved as .txt format:
test = "I use tensorflow for this data\n
I will be testing\n
The current tensorflow data
Please do mark that I am using tensorflow version 2.0 so I do not need to use tf.Session to initialize my variables
dataset = tf.data.TextLineDataset("test.txt")
dataset = dataset.map(lambda string: tf.string_split([string]).values)
dataset = dataset.padded_batch(2)
for x in dataset:
print(x.numpy())
Error that I received:
TypeError: padded_batch() missing 1 required positional argument: 'padded_shapes'
Expected output:
[[b'I' b'use' b'tensorflow' b'for' b'this' b'data']
[b'I' b'will' b'be' b'testing' b'unknown' b'unknown']]
[[b'The' b'current' b'tensorflow' b'data' b'unknown' b'unknown']]
How should I configure my padded_shapes and also padded_values? I wish to make the length of the tensor to be the same by insert "unknown" for each empty element. (This might be a little confused by above shows my expected results.)
Please note that tf.data.Dataset().dataset.padded_batch expects the shape of your inputs, and in your case, since you want the padded value to be "unknown" the padding value that you will use. Below is the code snipped you want to use.
dataset = tf.data.TextLineDataset("test.txt")
dataset = dataset.map(lambda string: tf.string_split([string]).values)
dataset = dataset.padded_batch(3, padded_shapes=[None], padding_values="unknown")
for x in dataset:
print(x.numpy())
# [[b'I' b'use' b'tensorflow' b'for' b'this' b'data']
# [b'I' b'will' b'be' b'testing' b'unknown' b'unknown']
# [b'The' b'current' b'tensorflow' b'data' b'unknown' b'unknown']]
I am trying to apply word2Vec and LSTM on a dataset that contains files' API trace log including API function calls and their parameters for a binary classification.
The data looks like:
File_ID, Label, API Trace log
1, M, kernel32 LoadLibraryA kernel32.dll
kernel32 GetProcAddress MZ\x90 ExitProcess
...
2, V, kernel32 GetModuleHandleA RPCRT4.dll
kernel32 GetCurrentThreadId d\x8B\x0D0 POINTER POINTER
...
The API trace including: module name, API function name, parameters (that separated by blank space)
Take first API trace of file 1 as example, kernel32 is the module name, LoadLibraryA is function name, kernel32.dll is parameter. Each API trace is separated by \n so that each line represents a API sequence information sequentially.
Firstly I trained a word2vec model based on the line sentence of all API trace log. There are about 5k API function calls, e.g. LoadLibraryA, GetProcAddress. However, because parameter value could be vary, the model becomes quite big (with 300,000 vocabulary) after including those parameters.
After that, I trained a LSTM by applying word2vec's embedding_wrights, the model structure looks like:
model = Sequential()
model.add(Embedding(output_dim=vocab_dim, input_dim=n_symbols, \
mask_zero=False, weights=[embedding_weights], \
trainable=False))
model.add(LSTM(dense_dim,kernel_initializer='he_normal', dropout=0.15,
recurrent_dropout=0.15, implementation=2))
model.add(Dropout(0.3))
model.add(Dense(1))
model.add(Activation('sigmoid'))
model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=epochs, batch_size=batch_size, callbacks=[early_stopping, parallel_check_cb])
The way I get embedding_weights is to create a matrix, for each vocabulary in word2vec model, map the index of the word in the model, to it's vector
def create_embedding_weights(model, max_index=0):
# dimensionality of your word vectors
num_features = len(model[model.vocab.keys()[0]])
n_symbols = len(model.vocab) + 1 # adding 1 to account for 0th index (for masking)
# Only word2vec feature set
embedding_weights = np.zeros((max(n_symbols + 1, max_index + 1), num_features))
for word, value in model.vocab.items():
embedding_weights[value.index, :] = model[word]
return embedding_weights
For training data, what I did is that for each word in API call, convert the actual word to the index of word2vec model so that it's consistent to the index in embedding_weights above. e.g. kernel32 -> 0, LoadLibraryA -> 1, kernel32.dll -> 2. GetProcAddress -> 4, MZ\x90 -> 5, ExitProcess ->6
So the train data for file 1 looks like [0, 1, 2, 3, 4, 5, 6]. Noted, I didn't do line split for each API trace. As a result, the model may not know where is the start and end of API trace? And the training accuracy of the model is pretty bad - accuracy is 50% :(
My question is that, when prepare the training and validation dataset, should I also split the line when mapping the actual words to their index? then The above training data would be changed to following, each API trace is separated by a line, and maybe padd the missing value to -1 which doesn't exist in word2vec's indexes.
[[0, 1, 2, -1]
[3, 4, 5, 6]]
Meanwhile I am using a very simple structure for training, while word2vec model is quite big, any suggestion on structure would also be appreciated.
I would at least split the trace lines in three:
Module (make a dictionary and an embedding)
Function (make a dictionary and an embedding)
Parameters (make a dictionary and an embedding - see details later)
Since this is a very specific application, I believe it would be best to keep the embeddings trainable (the whole point of the embeddings is to create meaningful vectors, and the meanings depend a lot on the model that is going to use them. Question: how did you create the word2vec model? From what data does it learn?).
This model would have more inputs. All of them as integers from zero to max dictionary index. Consider using mask_zero=True and padding all files to maxFileLines.
moduleInput = Input(maxFileLines,)
functionInput = Input(maxFileLines,)
For the parameters, I'd probably make a subsequence as if the list of parameters were a sentence. (Again, mask_zero=True, and pad up to maxNumberOfParameters)
parametersInput = Input(maxFileLines, maxNumberOfParameters)
Function and module embeddings:
moduleEmb = Embedding(.....mask_zero=True,)(moduleInput)
functionEmb = Embedding(.....mask_zero=True)(functionInput)
Now, for the parameters, I though of creating a sequence of sequences (maybe this is too much). For that, I first transfer the lines dimension to the batch dimension and work with only length = maxNumberOfParameters:
paramEmb = Lambda(lambda x: K.reshape(x,(-1,maxNumberOfParameters)))(parametersInput)
paramEmb = Embedding(....,mask_zero=True)(paramEmb)
paramEmb = Lambda(lambda x: K.reshape(x,(-1,maxFileLines,embeddingSize)))(paramEmb)
Now we concatenate all of them in the last dimension and we're ready to get into the LSTMs:
joinedEmbeddings = Concatenate()([moduleEmb,functoinEmb,paramEmb])
out = LSTM(...)(joinedEmbeddings)
out = ......
model = Model([moduleInput,functionInput,parametersInput], out)
How to prepare the inputs
With this model, you need three separate inputs. One for the module, one for the function and one for the parameters.
These inputs will contain only indices (no vectors). And they don't need a previous word2vec model. Embeddings are word2vec transformers.
So, get the file lines and split. First we split by commas, then we split the API calls by spaces:
import numpy as np
#read the file
loadedFile = open(fileName,'r')
allLines = [l.strip() for l in loadedFile.readlines()]
loadedFile.close()
#split by commas
splitLines = []
for l in allLines[1:]: #use 1 here only if you have headers in the file
splitLines.append (l.split(','))
splitLines = np.array(splitLines)
#get the split values and separate ids, targets and calls
ids = splitLines[:,0]
targets = splitLines[:,1]
calls = splitLines[:,2]
#split the calls by space, adding dummy parameters (spaces) to the max length
splitCalls = []
for c in calls:
splitC = c.strip().split(' ')
#pad the parameters (space for dummy params)
for i in range(len(splitC),maxParams+2):
splitC.append(' ')
splitCalls.append(splitC)
splitCalls = np.array(splitCalls)
modules = splitCalls[:,0]
functions = splitCalls[:,1]
parameters = splitCalls[:,2:] #notice the parameters have an extra dimension
Now lets make the indices:
modIndices, modCounts = np.unique(modules,return_counts=True)
funcIndices, funcCounts = np.unique(functions,return_counts=True)
#for de parameters, let's flatten the array first (because we have 2 dimensions)
flatParams = parameters.reshape((parameters.shape[0]*parameters.shape[1],))
paramIndices, paramCounts = np.unique(flatParams,return_counts=True)
These will create a list of unique words and get their counts. Here you can customize which words you're going to group in "another word" class. (Maybe based on the counts, if the count is too little, make it an "another word").
Let's then make the dictionaries:
def createDic(uniqueWords):
dic = {}
for i,word in enumerate(uniqueWords):
dic[word] = i + 1 # +1 because we want to reserve the zeros for padding
return dic
Just take care with the parameters, because we used a dummy space there:
moduleDic = createDic(modIndices)
funcDic = createDic(funcIndices)
paramDic = createDic(paramIndices[1:]) #make sure the space got the first position here
paramDic[' '] = 0
Well, now we just replace the original values:
moduleData = [moduleDic[word] for word in modules]
funcData = [funcDic[word] for word in functions]
paramData = [[paramDic[word] for word in paramLine] for paramLine in parameters]
Pad them:
for i in range(len(moduleData),maxFileLines):
moduleData.append(0)
funcData.append(0)
paramData.append([0] * maxParams)
Do this for every file, and store in a list of files:
moduleTrainData = []
functionTrainData = []
paramTrainData = []
for each file do the above and:
moduleTrainData.append(moduleData)
functionTrainData.append(funcData)
paramTrainData.append(paramData)
moduleTrainData = np.asarray(moduleTrainData)
functionTrainData = np.asarray(functionTrainData)
paramTrainData = np.asarray(paramTrainData)
That's all for the inputs.
model.fit([moduleTrainData,functionTrainData,paramTrainData],outputLabels,...)
How can one access to the learned weights of a DNN saved as following:
lstm_network_output.save(model_path)
The weights/parameters of a network can be accessed by calling ‘lstm_network_output.parameters’ which returns a list of ‘Parameter’ variable objects. The value of a Parameter can be obtained using ‘value’ property of the Parameter object in the form of a numpy array. The value of the Parameter can be updated by ‘.value = ’.
If you used name= properties in creating your model, you can also identify layers by name. For example:
model = Sequential([Embedding(300, name='embed'), Recurrence(LSTM(500)), Dense(10)])
E = model.embed.E # accesses the embedding matrix of the embed layer
To know that the parameter is .E, please consult the docstring of the respective function (e.g. help(Embedding)). (In Dense and Convolution, the parameters would be .W and .b.)
The pattern above is for named layers, which are created using as_block(). You can also name intermediate variables, and access them in the same way. E.g.:
W = Parameter((13,42), init=0, name='W')
x = Input(13)
y = times(x, W, name='times1')
W_recovered = y.times1.W
# e.g. check the shape to see that they are the same
W_recovered.shape # --> (13, 42)
W.shape # --> (13, 42)
Technically, this will search all parameters that feed y. In case of a more complex network, you may end up having multiple parameters of the same name. Then an error will be thrown due to the ambiguity. In that case, you must work the .parameters tuple mentioned in Anna's response.
This python code worked for me to visualize some weights:
import numpy as np
import cntk as C
dnnFile = C.cntk_py.Function.load('Models\ConvNet_MNIST_5.dnn') # load model from MS example
layer8 = dnnFile.parameters()[8].value()
filter_num = 0
sliced = layer8.asarray()[ filter_num ][ 0 ] # shows filter works on input image
print(sliced)