Can't see graph using torch.utils.tensorboard - tensorflow

I'm trying to get used to tensorboard, and I code my models using pytorch.
However when I try to see my model using the add_graph() function, I've got this:
With this as the test code:
import numpy as np
import torch
import torchvision.transforms as transforms
import torch.nn as nn
import torch.optim as optim
from torch.utils.tensorboard import SummaryWriter
class Net(nn.Module):
def __init__(self):
super(Net, self).__init__()
self.linear = nn.Linear(2, 1)
def forward(self, x):
x = self.linear(x)
return x
writer = SummaryWriter('runs_pytorch/test')
net = Net()
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
writer.add_graph(net, torch.zeros([4, 2], dtype=torch.float))
writer.close()
On the other hand, if I try to see a graph using TensorFlow, everything seems fine:
with this as the test code this time:
import tensorflow as tf
tf.Variable(42, name='foo')
w = tf.summary.FileWriter('runs_tensorflow/test')
w.add_graph(tf.get_default_graph())
w.flush()
w.close()
In case you are wondering, I'm using this command to start tensorboard:
tensorboard --logdir runs_pytorch
Something I noticed is that when I use it on the directory allocated for my tensorflow test, I've got the usual message with the address, but if I do the same thing with --logdir runs_pytorch I've got something more:
W1010 15:19:24.225109 15308 plugin_event_accumulator.py:294] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events. Overwriting the graph with the newest event.
W1010 15:19:24.226075 15308 plugin_event_accumulator.py:322] Found more than one "run metadata" event with tag step1. Overwriting it with the newest event.
I'm on windows, I tried on different browsers (chrome, firefox...).
I have tensorflow 1.14.0, torch 1.2.0, Python 3.7.3
Thank you very much for your help, it's driving me crazy!

There are two ways to solve it:
1. update PyTorch to 1.3.0 and above:
conda way:
conda install pytorch torchvision cudatoolkit=9.2 -c pytorch
pip way:
pip3 install torch==1.3.0+cu92 torchvision==0.4.1+cu92 -f https://download.pytorch.org/whl/torch_stable.html
2. install tensorboardX instead:
uninstall tensorboard:
if your tensorboard is installed by pip:
pip uninstall tensorboard
if your tensorboard is installed by anaconda:
conda uninstall tensorboard
install tensorboardX
pip install tensorboardX
when writing script,
change
from torch.utils.tensorboard import SummaryWriter
to
from tensorboardX import SummaryWriter

This might have been caused by this known problem, and it seems that it was solved in pytorch 1.3 which was realeased yesterday - check out Bug Fixes in the release notes.

Related

Run the code tensorflow 1.15 in tensorflow 2.7

I want to run code that uses tensorflow == 1.15 but has tensorflow 2.7 installed on my system. According to Tensorflow at this address https://www.tensorflow.org/guide/migrate/migrate_tf2, I use the following lines so that I can run the code without changing in tensorflow 2.7:
import tensorflow.compat.v1 as tf
tf.disable_v2_behavior()
But I do not know what to do instead of the following lines in my code!
Because, according to Tensorflow "You can still run unmodified TF1.x code (except for contrib) against TF2 binary installations"
l2_reg = tf.contrib.layers.l2_regularizer(scale=self.beta)
xavier = tf.contrib.layers.xavier_initializer()
please help
Tf.contrib is deprecated in Tensorflow 2.x
Replace
tf.contrib.layers.l2_regularizer
with
tf.compat.v1.keras.regularizers.l2
Replace
tf.contrib.layers.xavier_initializer
With
tf.compat.v1.keras.initializers.glorot_normal

how to use CRF in tensorflow keras?

The code is like this:
import tensorflow as tf
from keras_contrib.layers import CRF
from tensorflow import keras
def create_model(max_seq_len, adapter_size=64):
"""Creates a classification model."""
# adapter_size = 64 # see - arXiv:1902.00751
# create the bert layer
with tf.io.gfile.GFile(bert_config_file, "r") as reader:
bc = StockBertConfig.from_json_string(reader.read())
bert_params = map_stock_config_to_params(bc)
bert_params.adapter_size = adapter_size
bert = BertModelLayer.from_params(bert_params, name="bert")
input_ids = keras.layers.Input(shape=(max_seq_len,), dtype='int32', name="input_ids")
# token_type_ids = keras.layers.Input(shape=(max_seq_len,), dtype='int32', name="token_type_ids")
# output = bert([input_ids, token_type_ids])
bert_output = bert(input_ids)
print("bert_output.shape: {}".format(bert_output.shape)) # (?, 100, 768)
crf = CRF(len(tag2idx))
logits = crf(bert_output)
model = keras.Model(inputs=input_ids, outputs=logits)
model.build(input_shape=(None, max_seq_len))
# load the pre-trained model weights
load_stock_weights(bert, bert_ckpt_file)
# freeze weights if adapter-BERT is used
if adapter_size is not None:
freeze_bert_layers(bert)
model.compile('adam', loss=crf.loss_function, metrics=[crf.accuracy])
model.summary()
return model
I am using tensorflow keras and also use keras_contrib package, to do NER. it seems the tensorflow keras package does not work well with keras_contrib package.
The Traceback information is listed below:
Traceback (most recent call last):
File "F:/_gitclone3/bert_examples/bert_ner_example_eval.py", line 120, in <module>
model = create_model(max_seq_len, adapter_size=adapter_size)
File "F:/_gitclone3/bert_examples/bert_ner_example_eval.py", line 101, in create_model
logits = crf(bert_output)
File "C:\Users\yuexiang\Anaconda3\lib\site-packages\keras\engine\base_layer.py", line 443, in __call__
previous_mask = _collect_previous_mask(inputs)
File "C:\Users\yuexiang\Anaconda3\lib\site-packages\keras\engine\base_layer.py", line 1311, in _collect_previous_mask
mask = node.output_masks[tensor_index]
AttributeError: 'Node' object has no attribute 'output_masks'
How do I use CRF with tensorflow keras?
I run into a similar problem and spent a lot of time trying to get things to work. Here's what worked for me using python 3.6.5:
Seqeval:
pip install seqeval==0.0.5
Keras:
pip install keras==2.2.4
Keras-contrib (2.0.8):
git clone https://www.github.com/keras-team/keras-contrib.git
cd keras-contrib
python setup.py install
TensorFlow:
pip install tensorflow==1.14.0
Do pip list to make sure you have actually installed those versions (eg pip seqeval may automatically update your keras)
Then in your code import like so:
from keras.models import *
from keras.layers import LSTM, Embedding, Dense, TimeDistributed, Dropout, Bidirectional, Input
from keras_contrib.layers import CRF
#etc.
Hope this helps, good luck!
You can try tensorflow add-ons.(If you are using tensorflow version 2).
You can try tf-crf-layer (if you are using tensorflow==1.15.0)
They have it mentioned on their README.
git clone https://www.github.com/keras-team/keras-contrib.git
cd keras-contrib
python convert_to_tf_keras.py
USE_TF_KERAS=1 python setup.py install
I have gone through the possible solutions, mentioning which worked for me:
Install tf2crf (https://pypi.org/project/tf2crf/): It provides a simple CRF layer for TensorFlow 2 keras.
Use TensorFlow SIG Addons: ( https://www.tensorflow.org/addons/api_docs/python/tfa/layers/CRF): It provides the functionality that is not available in core TensorFlow.

Colab Kernel Restarts Whenever Loading a Model From Tensorflow-hub

I wanted to try out the embeddings provided in tensorflow-hub, the 'universal-sentence-encoder' to be specific. I tried the examples provided (https://colab.research.google.com/github/tensorflow/hub/blob/master/examples/colab/semantic_similarity_with_tf_hub_universal_encoder.ipynb)
and it worked fine. So I tried to do the same with 'multilingual' model but every time the multilingual model is loaded, the colab kernel fails and restarts. What is the problem and how can I get around this?
import tensorflow as tf
import tensorflow_hub as hub
import matplotlib.pyplot as plt
import numpy as np
import os
import pandas as pd
import re
import seaborn as sns
import tf_sentencepiece
import sentencepiece
# Import the Universal Sentence Encoder's TF Hub module
embed = hub.Module("https://tfhub.dev/google/universal-sentence-encoder-multilingual/1") // This is where the kernel dies.
print("imported model")
# Compute a representation for each message, showing various lengths supported.
word = "코끼리"
sentence = "나는 한국어로 쓰여진 문장이야."
paragraph = (
"동해물과 백두산이 마르고 닳도록. "
"하느님이 보우하사 우리나라 만세~")
messages = [word, sentence, paragraph]
# Reduce logging output.
tf.logging.set_verbosity(tf.logging.ERROR)
with tf.Session() as session:
session.run([tf.global_variables_initializer(), tf.tables_initializer()])
message_embeddings = session.run(embed(messages))
for i, message_embedding in enumerate(np.array(message_embeddings).tolist()):
print("Message: {}".format(messages[i]))
print("Embedding size: {}".format(len(message_embedding)))
message_embedding_snippet = ", ".join(
(str(x) for x in message_embedding[:3]))
print("Embedding: [{}, ...]\n".format(message_embedding_snippet))
I had similar issues with the multilingual sentence encoder. I resolved it by specifying tensorflow version to 1.14.0 and tf-sentencepiece to 0.1.83, so before running your code in colab try:
!pip3 install tensorflow==1.14.0
!pip3 install tensorflow-hub
!pip3 install sentencepiece
!pip3 install tf-sentencepiece==0.1.83
I was able to replicate your problem in colab and this solution loaded the model correctly:
It seems to be a compatibility problem between sentencepiece and tensorflow, check for updates on this issue here.
Let us know how it goes. Best of luck and I hope this helps.
EDIT: If tensorflow version 1.14.0 does not work, change it to 1.13.1. This problem should be resolved once compatibility between tensorflow and sentencepiece is figured it out.

I am not able to import resnet from keras.applications module

I'm unable to import this module
import keras.applications.resnet
ModuleNotFoundError
in ()
----> 1 import keras.applications.resnet
ModuleNotFoundError: No module named 'keras.applications.resnet'
keras resnet link
Keras team hasn't included resnet, resnet_v2 and resnext in the current module, they will be added from Keras 2.2.5, as mentioned here.
For a workaround, you can use keras_applications module directly to import all ResNet, ResNetV2 and ResNeXt models, as given below
from keras_applications.resnet import ResNet50
Or if you just want to use ResNet50
from keras.applications.resnet50 import ResNet50
Alternatively, you can always build from source as mentioned here.
try to use
from tensorflow.keras.applications.resnet50 import ResNet50
Found a workaround to use ResNeXt in Keras 2.2.4 here.
ResNeXt50() function needs 4 more arguments: backend, layers, models and utils.
import keras
from keras_applications.resnext import ResNeXt50
model = ResNeXt50(weights='imagenet',
backend=keras.backend,
layers=keras.layers,
models=keras.models,
utils=keras.utils)
In Keras there are multiple flavours of ResNet, you will have to specify the version of ResNet that you want e.g. You wish to load the ResNet50.
Use
from keras.applications import ResNet50
Edit 2 This is the list you get when you use dir() command on applications
['DenseNet121', 'DenseNet169', 'DenseNet201', 'InceptionResNetV2', 'InceptionV3', 'MobileNet', 'MobileNetV2', 'NASNetLarge', 'NASNetMobile', 'ResNet50', 'VGG16', 'VGG19', 'Xception', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'absolute_import', 'backend', 'densenet', 'division', 'inception_resnet_v2', 'inception_v3', 'keras_applications', 'keras_modules_injection', 'layers', 'mobilenet', 'mobilenet_v2', 'models', 'nasnet', 'print_function', 'resnet50', 'utils', 'vgg16', 'vgg19', 'xception'], the models visible here can be laoded like this, There are some models like ResNet101 missing here, let me see if I can come up with a way to fix this.
Edit Proof that this works too
To see all the available versions of the Resnet models, visit https://keras.io/applications/#resnet
There is a python package named 'keras-resnet' which has ResNet50, ResNet101, ResNet152 and many more variants of ResNet. (https://pypi.org/project/keras-resnet/)
Installation is also quite easy. Just type
pip install keras-resnet
It will install this module and then use it like:
from keras_resnet.models import ResNet50, ResNet101, ResNet152
backbone = ResNet50(inputs=image_input, include_top=False, freeze_bn=True)
C2, C3, C4, C5 = backbone.outputs # this will give you intermediate
# outputs of four blocks of resnet if you want to merge low and high level features
I am using backbones from this module and is working fine for me!
Check the versions:
pip list | grep Keras
If it's already installed, uninstall and upgrade:
pip uninstall Keras
pip install Keras==2.3.1
pip uninstall Keras-Applications
pip install Keras-Applications==1.0.8
pip uninstall Keras-Preprocessing
pip install Keras-Preprocessing==1.1.0
from keras.applications.resnet
import ResNet101
tf.keras.backend.clear_session
model=VGG19()
model.summary()
tf.keras.utils.plot_model(model,show_shapes=True)
visualkeras.layered_view(model,legend=True)
Before running
tensorflow.keras.applications.resnet50 import ResNet50
you need to run
from tensorflow import keras

Reproducible results with keras

How can I get reproducible results with keras? I followed these steps but I am still getting different results every time I run the Jupyter notebook. I also tried setting shuffle=False when calling model.fit().
My configuration:
conda 4.3.25
keras 2.0.6 with tensorflow backend
tensorflow-gpu 1.2.1
python 3.5
windows 10
See the answer I posted at another question. The main idea is to: first, disable the GPU. And then, seed the libraries like "numpy, random, etc". To sum up, including the code below at the beginning of your code may help solve your problem.
import os
os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"] = ""
import numpy as np
import tensorflow as tf
import random as rn
from keras import backend as K
sd = 1
np.random.seed(sd)
rn.seed(sd)
os.environ['PYTHONHASHSEED']=str(sd)
config = tf.ConfigProto(intra_op_parallelism_threads=1,inter_op_parallelism_threads=1)
tf.set_random_seed(sd)
sess = tf.Session(graph=tf.get_default_graph(), config=config)
K.set_session(sess)