When should I use tf.contrib.slim and when tf.layers?
As defined in https://www.tensorflow.org/api_docs/python/tf/contrib the contrib module contains "volatile or experimental code".
Generally speaking, the module tf.contrib contains contributed code. This code typically requires some additional tests and may encounter some significant changes before it is finally integrated into the TensorFlow core. In particular, this code is not supported by the Tensorflow team and may be modified or completely removed at any time without guarantees.
For this reason, in general I prefer to use tf.layers since it is more stable in terms of code support, but obviously some implementations intf.contrib can be useful sometimes (i.e. when there are new kind of layers, optimizers etc. and you cannot or don't want to write them yourself, sometimes these libraries are quickly updated).
Related
I am putting a model into production and I am required to scan all dependencies (Pytorch and Numpy) beforehand via VeraCode Scan.
I noticed that the majority of the flaws are coming from test scripts and caffe2 modules in Pytorch and numpy.
Is there any way to build/install only part of these packages that I use in my application? (e.g. I won't use testing and caffe2 in the application so there's no need to have them in my PyTorch / Numpy source code)
1. PyInstaller
You could package your application using pyinstaller. This tool packages your app with Python and dependencies and use only the parts you need (simplifying, in reality it's hard to trace your package exactly so some other stuff would be bundled as well).
Also you might be in for some quirks and workarounds to make it work with pytorch and numpy as those dependencies are quite heavy (especially pytorch).
2. Use only PyTorch
numpy and pytorch are pretty similar feature-wise (as PyTorch tries to be compatible with it) hence maybe you could only use only of them which would simplify the whole thing further
3. Use C++
Depending on other parts of your app you may write it (at least neural network) in C++ using PyTorch's C++ frontend which is stable since 1.5.0 release.
Going this route would allow you to compile PyTorch's .cpp source code statically (so all dependencies are linked) which allows you for relatively small binary size (30Mb when compared to PyTorch's 1GB+), but requires a lot of work.
I have been working in TensorFlow for about a year now, and I am transitioning from TF 1.x to TF 2.0, and I am looking for some guidance on how to use the tf.keras.backend library in TF 2.0. I understand that the transition to TF 2.0 is supposed to remove a lot of redundancies in modeling and building graphs, since there were many ways to create equivalent layers in earlier TensorFlow versions (and I'm insanely grateful for that change!), but I'm getting stuck on understanding when to use tf.keras.backend, because the operations appear redundant with other TensorFlow libraries.
I see that some of the functions in tf.keras.backend are redundant with other TensorFlow libraries. For instance, tf.keras.backend.abs and tf.math.abs are not aliases (or at least, they're not listed as aliases in the documentation), but both take the absolute value of a tensor. After examining the source code, it looks like tf.keras.backend.abs calls the tf.math.abs function, and so I really do not understand why they are not aliases. Other tf.keras.backend operations don't appear to be duplicated in TensorFlow libraries, but it looks like there are TensorFlow functions that can do equivalent things. For instance, tf.keras.backend.cast_to_floatx can be substituted with tf.dtypes.cast as long as you explicitly specify the dtype. I am wondering two things:
when is it best to use the tf.keras.backend library instead of the equivalent TensorFlow functions?
is there a difference in these functions (and other equivalent tf.keras.backend functions) that I am missing?
Short answer: Prefer tensorflow's native API such as tf.math.* to thetf.keras.backend.* API wherever possible.
Longer answer:
The tf.keras.backend.* API can be mostly viewed as a remnant of the keras.backend.* API. The latter is a design that serves the "exchangeable backend" design of the original (non-TF-specific) keras. This relates to the historical aspect of keras, which supports multiple backend libraries, among which tensorflow used to be just one of them. Back in 2015 and 2016, other backends, such as Theano and MXNet were quite popular too. But going into 2017 and 2018, tensorflow became the dominant backend of keras users. Eventually keras became a part of the tensorflow API (in 2.x and later minor versions of 1.x). In the old multi-backend world, the backend.* API provides a backend-independent abstraction over the myriad of supported backend. But in the tf.keras world, the value of the backend API is much more limited.
The various functions in tf.keras.backend.* can be divided into a few categories:
Thin wrappers around the equivalent or mostly-equivalent tensorflow native API. Examples: tf.keras.backend.less, tf.keras.backend.sin
Slightly thicker wrappers around tensorflow native APIs, with more features included. Examples: tf.keras.backend.batch_normalization, tf.keras.backend.conv2d(https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/keras/backend.py#L4869). They often perform proprocessing and implement other logics, which make your life easier than using native tensorflow API.
Unique functions that don't have equivalent in the native tensorflow API. Examples: tf.keras.backend.rnn, tf.keras.backend.set_learning_phase
For category 1, use native tensorflow APIs. For categories 2 and 3, you may want to use the tf.keras.backend.* API, as long as you can find it in the documentation page: https://www.tensorflow.org/api_docs/python/, because the documented ones have backward compatibility guarantees, so that you don't need to worry about a future version of tensorflow removing it or changing its behavior.
I want to know if CNTK dead? Release notes on GitHub dated 03/31/2019: "Today’s 2.7 release will be the last main release of CNTK." I've spent months developing software using CNTK and now it appears to be a waste of time and money. I've search for an answer on numerous sites and still no answer. stackoverflow is one of the sites recommend by Microsoft.
From KedengMS, one of the maintainers for CNTK. Reposted from github.
Thanks for all the CNTK supporters, and I am privileged to have worked
on it, and learned a lot in the process. You can continue to use CNTK
for training and inference in the way it currently is, as other
Microsoft internal teams that still runs old models even in
BrainScript or NDL. Stopping adding new features does not mean CNTK is
no longer open source, it just means that going forward, there will be
no new GPU support (say, CUDA 11+), and no major new features added.
For different user scenarios, I think you may have different choices:
Deep learning newcomers: IMO CNTK is still a good entry to understand basics of deep learning, if you found CNTK
documents/tutorials/examples useful. Once you learnt the basic, it
won't be too hard to switch between frameworks. However, the DL field
is changing rapidly and CNTK has already lagged behind in a lot of
ways, so if you need more advanced features like dynamic graph,
PyTorch would be a better choice.
Model maintainers: If you already have CNTK models working, and to maintain it just means training with new data, you can continue to use
CNTK the way you currently use it. Actually, teams inside Microsoft
are doing this too. If there are serious bugs preventing productivity,
they still will be fixed. For inference, you can continue to use CNTK
C/C++/Python/C#/Java APIs, or you may export CNTK models in ONNX
format, and use ONNX Runtime or ORT as a slimmer and faster inference
engine. You'll be surprised to find how much faster it is comparing to
CNTK, and how slimmer the setup is (forget about OpenMPI when you just
need inference!). ORT currently provides C/C++/Python/C# interfaces.
Model builders: If you have CNTK model, and want to use features that are not currently supported in CNTK, please consider switch to
other frameworks like TensorFlow/PyTorch/etc. Our team has done lots
of data reader work inside PyTorch to ensure teams in Microsoft can
switch from CNTK to PyTorch. Besides, we are also in the process of
migrating CNTK specific distributed trainer like BMUF to PyTorch.
Hopefully you'll find that useful too when migrating your model.
The good thing about open source is that the community can continue to
fork/evolve if needed, unlike other Microsoft products that only ship
binaries (Win7 I am looking at you).
It seems that tf.Layer modules come in two flavours: functions and classes. I normally use the functions directly (e.g, tf.layers.dense) but I'd like to know how to use classes directly (tf.layers.Dense). I've started experimenting with the new eager execution mode in tensorflow and I think using classes are going to be useful there as well but I haven't seen good examples in the documentation. Is there any part of TF documentation that shows how these are used?
I guess it would make sense to use them in a class where these layers are instantiated in the __init__ and then they're linked in the __call__ method when the inputs and dimensions are known?
Are these tf.layer classes related to tf.keras.Model? Is there an equivalent wrapper class for using tf.layers?
Update: for eager execution there's tfe.Network that must be inherited. There's an example here
tf.layers and tf.keras.layer classes are generally interchangeable and in fact at head (and thus by the next release - 1.9), the former actually inherits from the latter.
TensorFlow is moving towards consolidating on tf.keras APIs for constructing models as that makes state ownership more explicit (e.g., parameters are "owned" by the Layer object, as opposed to the functional style where all model parameters are put in a "collection" associated with the complete graph). This style works well for both eager execution and graph construction (support for eager execution is improving with every release). I'd recommend using tf.keras.layers and tf.keras.Model.
Some examples that you may find useful:
MNIST in the tensorflow/models repository
The programmer's guide
Other eager execution samples (where the exact same model definition works for both graph execution and eager execution).
Not all existing TensorFlow examples have been moved to this style, but they slowly will.
Hope that helps.
I want to study on the research of deep learning, but I don't know which framwork should I choice between TensorFlow and PaddlePaddle. who can make a contrast between the two frameworks? which one is better? especially in the running efficiency of CPU
It really depends what you are shooting for...
If you plan on training, CPU is not going to work well for you. Use colab or kaggle.
Assuming you do get a GPU, it depends if you want to focus on classification or object detection.
If you focus on classification, Keras is probably the easiest to work with or pytorch if you want some advanced stuff and to be able to change things.
If you plan on object detection, things are getting complicated... Inference is reasonably easy but training is complicated. There are actually 4 platforms you should consider:
Tensorflow - powerful but very difficult to work with. If you do not use Keras (and for OD you usually can't), you need to preprocess the dataset into tfrecords and it is a pain. The OD Api has very cryptic messages and it is very sensitive to the combination of tf version and api version. On the other hand, cool models like efficientdet are more or less easy to use.
MMdetection - very powerful framework, has lots of advanced models and once you understand how to work with it, you can easily work with and of the models it supports. Downside is that some models are slow to arrive (efficientdet, for example)
paddlepaddle - if you know Chinese, this should work ok, maybe. The documentation is a bit behind and usually requires lots of improvisation. Basically it is similar to mmdetection just with a few unique models and a few missing models.
detectron2 - I didn't work with this one, but it seems to support only a few models.
You probably need first to define for yourself what do you want to do and then choose.
Good luck!
It is not that trivial. Some models run faster with one kind of framework others with another. Furthermore, it depends on the hardware as well. See this blog. If inference is your only concern, then you can develop your model in any of the popular frameworks like TensorFlow, PyTorch, etc. In the end convert your model to ONNX format and benchmark its performance with DNN-Bench to choose the best inference engine for your application.