From what I understand, using feed_dict is a computationally expensive process and should be avoided according to this article. Tensorflow's input pipelines are supposedly better.
All mini-batch gradient descent tutorials that I've found are implemented with feed_dict. Is there a way to use input pipelines and minibatch gradient descent?
If you are just making a small model, you will do fine with feed_dict. Many large models have been trained with the feed_dict method in the past. If you are scaling to a very deep convnet with a large dataset or something, you may want to use tf.data and the dataset pipeline, probably with the dataset serialized to a .tfrecord file so that the data can be pre-fetched to the GPU to reduce idle time. These optimizations are worthwhile with large models, and if you would really like to learn the API, visit the quickstart guide and this helpful Medium article on getting started with the API.
Related
I am interested in building a yolo detector with trained on multiple datasets where each dataset has it own detection head. It is a multi-task learning approach. I am not sure how to convert the yolo detector architecture to support multiple head.
I came across the following projects, however I need your help to implement similar approach.
https://github.com/xingyizhou/UniDet
https://link.springer.com/chapter/10.1007/978-981-16-6963-7_27
This approach has some difficulties. First, in article you sent they use two-stage detection model with separate classification "branches". In the same time YOLO is one-stage detector and is fullyconvolutional, that means there are no fullyconnected layers, and class predictions (1d) are taking from the whole 3d-tensor (see the image).
You can take a look at YOLO9000 paper, the model was trained on detection and classification datasets at the same time - only loss function was changing.
I am doing machine learning research where I explore non backpropagation options for training neural networks. I have implemented a genetic algorithm for training of keras models but I would like to make some improvements.
Currently I'm training the model by extracting the parameters with model.get_weights(), modifying them in a genetic algorithm, and then setting the weights again, with model.set_weights(), for loss calculation. This is being done by passing the network to a training function that performs the training and returns a history as such:
history = genetic_algorithm(model, *args).
I would like to act within the keras interface if possible so that I could compile the model using my training function somehow and then simply train with model.fit(). Is this possible? I saw this post regarding creating custom optimizers but it seems to be to constrained to allow a genetic algorithm.
This question is related to another question I have asked. I split them up to make each question more concise.
I followed all the steps mentioned in the article:
https://stackabuse.com/tensorflow-2-0-solving-classification-and-regression-problems/
Then I compared the results with Linear Regression and found that the error is less (68) than the tensorflow model (84).
from sklearn.linear_model import LinearRegression
logreg_clf = LinearRegression()
logreg_clf.fit(X_train, y_train)
pred = logreg_clf.predict(X_test)
print(np.sqrt(mean_squared_error(y_test, pred)))
Does this mean that if I have large dataset, I will get better results than linear regression?
What is the best situation - when I should be using tensorflow?
Answering your first question, Neural Networks are notoriously known for overfitting on smaller datasets, and here you are comparing the performance of a simple linear regression model with a neural network with two hidden layers on the testing data set, so it's not very surprising to see that the MLP model falling behind (assuming that you are working with relatively a smaller dataset) the linear regression model. Larger datasets will definitely help neural networks in learning more accurate parameters and generalize the phenomena well.
Now coming to your second question, Tensorflow is basically a library for building deep learning models, so whenever you are working on a deep learning problem like image recognition, Natural Language Processing, etc. you need massive computational power and will be processing a ton of data to train your models, and this is where TensorFlow becomes handy, it offers you GPU support which will significantly boost your training process which otherwise becomes practically impossible. Moreover, if you are building a product that has to be deployed in a production environment for it to be consumed, you can make use of TensorFlow Serving which helps you to take your models much closer to the customers.
I am setting up an object detection pipeline based on recently released tensorflow object detection API. I am using the arXiv as guidance. I am looking to understand the below for training on my own dataset.
It is not clear how they selected the learning rate schedules and how that would change based on the number of GPUs available for training. How do the training rate schedule change based on number of GPU's available for training? The paper mentions 9 GPUs are used. How should I change the training rate if I only want to use 1 GPU?
The released sample training config file for Pascal VOC using Faster R-CNN has initial learning rate = 0.0001. This is 10x lower than what was published in the original Faster-RCNN paper. Is this due to an assumption on the number of GPU's available for training or due to a different reason?
When I start training from the COCO detection checkpoint, how should the training loss decrease? Looking at tensorboard, on my dataset training loss is low - between 0.8 to 1.2 per iteration (with batch size of 1). Below image shows the various losses from tensorboard. . Is this expected behavior?
For questions 1 and 2: our implementation differs in a few small details compared to the original paper and internally we train all of our detectors with asynchronous SGD with ~10 GPUs. Our learning rates are calibrated for this setting (which you will also have if you decide to train via Cloud ML Engine as in the Pets walkthrough). If you use another setting, you will have to do a bit of hyperparameter exploration. For a single GPU, leaving the learning rate alone probably won't hurt performance, but you may be able to get faster convergence by increasing it.
For question 3: Training losses decrease erratically and you can only see the decrease if you smooth the plots quite a bit over time. Moreover, it's hard to explicitly say how well you are doing with respect to eval metrics just by looking at the training losses. I recommend looking at the mAP plots over time as well as the image visualizations to really get an idea of whether your model has "lifted off".
Hope this helps.
Has anyone implement the FRCNN for TensorFlow version?
I found some related repos as following:
Implement roi pool layer
Implement fast RCNN based on py-faster-rcnn repo
but for 1: assume the roi pooling layer works (I haven't tried), and there are something need to be implemented as following:
ROI data layer e.g. roidb.
Linear Regression e.g. SmoothL1Loss
ROI pool layer post-processing for end-to-end training which should convert the ROI pooling layer's results to feed into CNN for classifier.
For 2: em...., it seems based on py-faster-rcnn which based on Caffe to prepared pre-processing (e.g. roidb) and feed data into Tensorflow to train the model, it seems weird, so I may not tried it.
So what I want to know is that, will Tensorflow support Faster RCNN in the future?. If not, do I have any mis-understand which mentioned above? or has any repo or someone support that?
Tensorflow has just released an official Object Detection API here, that can be used for instance with their various slim models.
This API contains implementation of various Pipelines for Object Detection, including popular Faster RCNN, with their pre-trained models as well.