I am trying to build a NER model in Keras using ELMO Embeddings. SO I stumped across this tutorial and started implementing. I got lots of errors and some of them are as:
import tensorflow as tf
import tensorflow_hub as hub
from keras import backend as K
sess = tf.Session()
K.set_session(sess)
elmo_model = hub.Module("https://tfhub.dev/google/elmo/2", trainable=True)
sess.run(tf.global_variables_initializer())
sess.run(tf.tables_initializer())
def ElmoEmbedding(x):
return elmo_model(inputs={"tokens": tf.squeeze(tf.cast(x, tf.string)),
"sequence_len": tf.constant(batch_size*[max_len])},signature="tokens",as_dict=True)["elmo"]
input_text = Input(shape=(max_len,), dtype=tf.string)
embedding = Lambda(ElmoEmbedding, output_shape=(None, 1024))(input_text)
It gives me AttributeError: module 'tensorflow' has no attribute 'Session' . So if I comment out sess= code and run, it gives me AttributeError: module 'keras.backend' has no attribute 'set_session'.
Then again, Elmo code line is giving me RuntimeError: Exporting/importing meta graphs is not supported when eager execution is enabled. No graph exists when eager execution is enabled..
I have the following configurations:
tf.__version__
'2.3.1'
keras.__version__
'2.4.3'
import sys
sys.version
'3.8.3 (default, Jul 2 2020, 17:30:36) [MSC v.1916 64 bit (AMD64)]'
How can I use ELMO Embeddings in Keras Model?
You are using the old Tensorflow 1.x syntax but you have tensorflow 2 installed.
This is the new way to do elmo in TF2
Extracting ELMo features using tensorflow and convert them to numpy
Related
I am testing a simple model with the main to serving it . I am in colab environment.
executing the code
import numpy as np
from tensorflow import keras
model=tf.keras.Sequential([keras.layers.Dense(units=1,input_shape=[1])])
model.compile(optimizer='sgd', loss='mean_squared_error')
xs=np.array([-1.0,0.0,1.0,2.0,3.0,4.0],dtype=float)
ys=np.array([-3.0,-1.0,1.0,3.0,5.0,7.0],dtype=float)
model.fit(xs,ys,epochs=500,verbose=2)
tf.saved_model.simple_save(
keras.backend.get_session(),
export_path,
inputs={'input_image': model.input},
outputs={t.name:t for t in model.outputs})
I obtain the following error that I don't know how to fix:
<ipython-input-19-634675006b49> in <module>()
----> 1 tf.saved_model.simple_save(
2 keras.backend.get_session(),
3 export_path,
4 inputs={'input_image': model.input},
5 outputs={t.name:t for t in model.outputs})
AttributeError: module 'tensorflow._api.v2.saved_model' has no attribute 'simple_save'```
simple_save is deprecated in tensorflow v2 (LINK).
Try to use model.save(saving/path) instead
full documentation on model.save : https://www.tensorflow.org/tutorials/keras/save_and_load#save_the_entire_model
The code is like this:
import tensorflow as tf
from keras_contrib.layers import CRF
from tensorflow import keras
def create_model(max_seq_len, adapter_size=64):
"""Creates a classification model."""
# adapter_size = 64 # see - arXiv:1902.00751
# create the bert layer
with tf.io.gfile.GFile(bert_config_file, "r") as reader:
bc = StockBertConfig.from_json_string(reader.read())
bert_params = map_stock_config_to_params(bc)
bert_params.adapter_size = adapter_size
bert = BertModelLayer.from_params(bert_params, name="bert")
input_ids = keras.layers.Input(shape=(max_seq_len,), dtype='int32', name="input_ids")
# token_type_ids = keras.layers.Input(shape=(max_seq_len,), dtype='int32', name="token_type_ids")
# output = bert([input_ids, token_type_ids])
bert_output = bert(input_ids)
print("bert_output.shape: {}".format(bert_output.shape)) # (?, 100, 768)
crf = CRF(len(tag2idx))
logits = crf(bert_output)
model = keras.Model(inputs=input_ids, outputs=logits)
model.build(input_shape=(None, max_seq_len))
# load the pre-trained model weights
load_stock_weights(bert, bert_ckpt_file)
# freeze weights if adapter-BERT is used
if adapter_size is not None:
freeze_bert_layers(bert)
model.compile('adam', loss=crf.loss_function, metrics=[crf.accuracy])
model.summary()
return model
I am using tensorflow keras and also use keras_contrib package, to do NER. it seems the tensorflow keras package does not work well with keras_contrib package.
The Traceback information is listed below:
Traceback (most recent call last):
File "F:/_gitclone3/bert_examples/bert_ner_example_eval.py", line 120, in <module>
model = create_model(max_seq_len, adapter_size=adapter_size)
File "F:/_gitclone3/bert_examples/bert_ner_example_eval.py", line 101, in create_model
logits = crf(bert_output)
File "C:\Users\yuexiang\Anaconda3\lib\site-packages\keras\engine\base_layer.py", line 443, in __call__
previous_mask = _collect_previous_mask(inputs)
File "C:\Users\yuexiang\Anaconda3\lib\site-packages\keras\engine\base_layer.py", line 1311, in _collect_previous_mask
mask = node.output_masks[tensor_index]
AttributeError: 'Node' object has no attribute 'output_masks'
How do I use CRF with tensorflow keras?
I run into a similar problem and spent a lot of time trying to get things to work. Here's what worked for me using python 3.6.5:
Seqeval:
pip install seqeval==0.0.5
Keras:
pip install keras==2.2.4
Keras-contrib (2.0.8):
git clone https://www.github.com/keras-team/keras-contrib.git
cd keras-contrib
python setup.py install
TensorFlow:
pip install tensorflow==1.14.0
Do pip list to make sure you have actually installed those versions (eg pip seqeval may automatically update your keras)
Then in your code import like so:
from keras.models import *
from keras.layers import LSTM, Embedding, Dense, TimeDistributed, Dropout, Bidirectional, Input
from keras_contrib.layers import CRF
#etc.
Hope this helps, good luck!
You can try tensorflow add-ons.(If you are using tensorflow version 2).
You can try tf-crf-layer (if you are using tensorflow==1.15.0)
They have it mentioned on their README.
git clone https://www.github.com/keras-team/keras-contrib.git
cd keras-contrib
python convert_to_tf_keras.py
USE_TF_KERAS=1 python setup.py install
I have gone through the possible solutions, mentioning which worked for me:
Install tf2crf (https://pypi.org/project/tf2crf/): It provides a simple CRF layer for TensorFlow 2 keras.
Use TensorFlow SIG Addons: ( https://www.tensorflow.org/addons/api_docs/python/tfa/layers/CRF): It provides the functionality that is not available in core TensorFlow.
I am trying to transform my Keras model in the Google cloud console into a TPU model. Unfortunatelly I am getting an error as shown below. My minimal example is the following:
import keras
from keras.models import Sequential
from keras.layers import Dense, Activation
import tensorflow as tf
import os
model = Sequential()
model.add(Dense(32, input_dim=784))
model.add(Dense(32))
model.add(Activation('relu'))
model.compile(optimizer='rmsprop', loss='mse')
tpu_model = tf.contrib.tpu.keras_to_tpu_model(
model,
strategy=tf.contrib.tpu.TPUDistributionStrategy(
tf.contrib.cluster_resolver.TPUClusterResolver(TPU_WORKER)))
My output is:
Using TensorFlow backend.
Traceback (most recent call last):
File "cloud_python4.py", line 11, in <module>
tpu_model = tf.contrib.tpu.keras_to_tpu_model(AttributeError: module 'tensorflow.contrib.tpu' has no attribute 'keras_to_tpu_model'
The keras_to_tpu_model method seems experimental as indicated on the tensorflow website. Has it recently been removed? If so, how can I proceed to make use of TPUs to estimate my Keras model? If the keras_to_tpu_model method would be still available, why can I not invoke it?
I am assuming you defined you TPU_WORKER as below
import os
TPU_WORKER = ‘grpc://’ + os.environ[‘COLAB_TPU_ADDR’]
Instead of converting your model to TPU, build a distribution strategy. This is the method by which the batch will be distributed to the eight TPUs and how the loss from each will be calculated.
resolver = tf.contrib.cluster_resolver.TPUClusterResolver(TPU_WORKER)
tf.contrib.distribute.initialize_tpu_system(resolver)
strategy = tf.contrib.distribute.TPUStrategy(resolver)
With the strategy build and compile your model. This should work quite nicely for regression.
with strategy.scope():
model = Sequential()
model.add(Dense(32, input_dim=784))
model.add(Dense(32))
model.add(Activation('relu'))
model.compile(optimizer='rmsprop', loss='mse')
Import keras from tensorflow.
This is because tf.contrib.tpu.keras_to_tpu_model( )' requires a tensorflow version Model, not the keras version.
For example, use from tensorflow.keras.layers import Dense, Activation instead. And so on.
Is it possible to convert a keras model (h5 file of network architecture and weights) into a tensorflow model? Or is there an equivalent function to model.save of keras in tensorflow?
Yes, it is possible, because Keras, since it uses Tensorflow as backend, also builds computational graph. You just need to get this graph from your Keras model.
"Keras only uses one graph and one session. You can access the session
via: K.get_session(). The graph associated with it would then be:
K.get_session().graph."
(from fchollet: https://github.com/keras-team/keras/issues/3223#issuecomment-232745857)
Or you can save this graph in checkpoint format (https://www.tensorflow.org/api_docs/python/tf/train/Saver):
import tensorflow as tf
from keras import backend as K
saver = tf.train.Saver()
sess = K.get_session()
retval = saver.save(sess, ckpt_model_name)
By the way, since tensorflow 13 you can use keras right from it:
from tensorflow.python.keras import models, layers
I have a code in Keras (or its TF version). I want to have a CUDA code which is equivalence to it. Is there a way to get it?
I know that from Keras I can look at the basic graph topology using the following code:
# LSTM for sequence classification in the IMDB dataset
import numpy
from keras.datasets import imdb
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from keras.layers.embeddings import Embedding
from keras import backend as K
from keras.preprocessing import sequence
# fix random seed for reproducibility
numpy.random.seed(7)
# load the dataset but only keep the top n words, zero the rest
top_words = 5000
max_review_length = 500
# create the model
embedding_vecor_length = 32
model = Sequential()
model.add(Embedding(top_words, embedding_vecor_length, input_length=max_review_length))
model.add(LSTM(100))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
g = K.get_session().graph
# GIVES THE GRAPH TOPOLOGY!:
graph_def = g.as_graph_def()
Is there a way to have the .cc file that represent this code?
Thanks!
There is no functionality in TensorFlow to generate C++ CUDA source code from a graph, but the XLA framework supports ahead-of-time compilation, which generates efficient bytecode from your TensorFlow graph, which you can then execute on your CUDA-capable GPU.