Trying to find any Tensorflow/Keras implementation of the original BERT model trained using MLM/NSP. The official google and HuggingFace implementations are very complex and has so much of added functionalities. But I want to learn and implement BERT for just learning its working.
Any leads will be helpful?
As mentioned in the comment, you can try the following implementation of MLP-BERT TensorFlow. It's a simplified version and easy to follow comparatively.
Related
I am currently trying to quantize a bert-classifier model but am running into an error, I was wondering if this is even supported at the moment or not? For clarity I am asking if quantization is supported on the BERT Classifier super class in the tensorflow-model-garden? Thanks in advance for the help!
Quantizing the standard BERT classifier is probably not a good way to go, if you are interesting in running a BERT-like model on a resource constrained edge device (like a mobile phone). For your specific question, I believe the answer is 'no, quantization of the standard BERT is not supported.' However, a better answer is probably to use one of the smaller BERT-type models that have been created for the edge use case, such as MobileBERT:
https://github.com/google-research/google-research/tree/master/mobilebert
The above link includes scripts for fine-tuning and then converting to TF Lite format in order to run on device.
My goal is to test out Google's BERT algorithm in Google Colab.
I'd like to use a pre-trained custom model for Finnish (https://github.com/TurkuNLP/FinBERT). The model can not be found on TFHub library. I have not found a way to load model with Tensorflow Hub.
Is there a neat way to load and use a custom model with Tensorflow Hub?
Fundamentally: yes. Everyone can create the kind of models that TF Hub hosts, and I hope authors of interesting models do consider that.
For TF1 and the hub.Module format tailored to it, see
https://www.tensorflow.org/hub/tf1_hub_module#creating_a_new_module
For TF2 and its revised SavedModel format, see
https://www.tensorflow.org/hub/tf2_saved_model#creating_savedmodels_for_tf_hub
That said, a sophisticated model like BERT requires a bit of attention to export it with all bells and whistles, so it helps to have some tooling to build on. The BERT reference implementation for TF2 at https://github.com/tensorflow/models/tree/master/official/nlp/bert comes with an open-sourced export_tfhub.py script, and anyone can use that to export custom BERT instances created from that code base.
However, I understand from https://github.com/TurkuNLP/FinBERT/blob/master/nlpl_tutorial/training_bert.md#general-info that you are using Nvidia's fork of the original TF1 implementation of BERT. There are Hub modules created from the original research code, but the tooling to that end has not been open-sourced, and Nvidia doesn't seem to have added their own either.
If that's not changing, you'll probably have to resort to doing things the pedestrian way and get acquainted with their codebase and load their checkpoints into it.
I need to re-write some code from pyTorch to CNTK.
I know CNTK and deep learning basics quite well.
Is it easy to relate pyTorch and CNTK?
Do I need to be aware of some special things?
I did translate TensorFlow and CNTK codes before, and I found it easy.
But I know Tensorflow reasonably well ... but now I do not want to put effort in learning pyTorch.
One thing you can try is to export your model to ONNX format from PyTorch. Then you can load it in CNTK. See here for saving and here for loading. Note that this is still a very new effort and different toolkits have different degrees of support for ONNX.
Are there any resnet implementations in tensorflow? I came across a few (e.g. https://github.com/ry/tensorflow-resnet, https://github.com/xuyuwei/resnet-tf) but these implementations have some bugs (e.g. see the Issues section on the respective github page). I am looking to train imagenet using resnet and looking for tensorflow implementations.
There are some (50/101/152) in tensorflow:models/slim.
The example notebook shows how to get a pre-trained inception running, res-net is probably no different.
I implemented a cifar10 version of ResNet with tensorflow. The validation errors of ResNet-32, ResNet-56 and ResNet-110 are 6.7%, 6.5% and 6.2% respectively. (You can modify the number of layers easily as hyper-parameters.)
I tried to be friendly with new ResNet fan and wrote everything straightforward. You can run the cifar10_train.py file directly without any downloads.
https://github.com/wenxinxu/resnet_in_tensorflow
I implemented Resnet by use of ronnie.ai and keras. Both of tool are great.
While ronnie is more easy from scratch.
i am interested in trying things out with a spatial transformer network and I can't find any implementation of it in caffe or tensorflow, which are the only two libraries I'm interested in using. I have a pretty good grasp of tensorflow but was wondering if it would be straight forward to implement with the existing building blocks that tensorflow offers without having to do something too complicated like write a custom c++ module
Yes, it is very straight forward to setup the Tensorflow graph for a spatial transformer network with the existing API.
You can find an example implementation in Tensorflow here [1].
[1] https://github.com/daviddao/spatial-transformer-tensorflow
There is an implementation in caffe here. https://github.com/daerduoCarey/SpatialTransformerLayer
Tensorflow has a implementation of Spatial Transformer Network in the models repository - https://github.com/tensorflow/models/tree/master/research/transformer