Is there a direct BigTable connector for TensorFlow 2.x ?
I haven't found any clear track of such package in the master branches of the Github repos tensorflow and tensorflow_io.
Thanks !
Support on Bigtable was started on tfio v0.5.0 but was removed on v0.13.0. Here's the note:
BigTable has been broken from the beginning, and was never maintained.
In addition, there are lots of code duplication as it implment 6 datasets
in C++ with very minimal difference. Due to the recent changes in TF upsteram
API change, it gets harder and harder to even make it compile.
This PR removes BigTable from the code tree. The plan is to add BigTable in the future
with right implementation, ideally utilize the better maintained google cloud C++ big table API: https://github.com/googleapis/google-cloud-cpp/tree/master/google/cloud/bigtable
Check their commit log: https://github.com/tensorflow/io/commit/f08f7954631cd13b3ace059dbc05f0b71dcd857d
Related
I have hundreds of models, based on categories, projects,s, etc. Some of the models are heavily used while other models are not used very frequently.
How can I trigger a scale-up operation only in case needed (For the models that are not frequently used), instead of running hundreds of pods serving hundreds of models while most of them are not being used - which is a huge waste of computing resources.
What you are trying to do is to scale deployment to zero when these are not used.
K8s does not provide such functionality out of the box.
You can achieve it using Knative Pod Autoscaler.
Knative is probably the most mature solution available at the moment of writing this answer.
There are also some more experimental solutions like osiris or zero-pod-autoscaler you may find interesting and that may be a good fit for your usecase.
For example, will it eventually work? Does it work? What are the goals and plans? Where can we read about it.
Is tensorflow_transform a going concern for tf 2.0?
Absolutely! Development is ongoing. Issues are being actively discussed, PRs are being worked on and there have been several changes to the master branch this week.
will it eventually work? Does it work?
Yes it works now (in general at least). Perhaps if you are encountering some specific issue could ask a new question with what, specifically, isnt working for you.
What are the goals and plans? Where can we read about it.
The tensorflow team are really good at communicating plans via RFCs and doing development in the open. I am less familiar with work on tf-transform but all the signs are this is developed with the same culture. Check out:
the github repo
the official site
I am trying to understand what are the basic difference between Tensorflow Mirror Strategy and Horovod Distribution Strategy.
From the documentation and the source code investigation I found that Horovod (https://github.com/horovod/horovod) is using Message Passing Protocol (MPI) to communicate between multiple nodes. Specifically it uses all_reduce, all_gather of MPI.
From my observation (I may be wrong) Mirror Strategy is also using all_reduce algorithm (https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/distribute).
Both of them are using data-parallel, synchronous training approach.
So I am a bit confused how they are different? Is the difference only in implementation or there are other (theoretical) difference?
And how is the performance of mirror strategy compared to horovod?
Mirror Strategy has its own all_reduce algorithm which use remote procedural calls (gRPC) under the hood.
Like you mentioned Horovod uses MPI/GLOO to communicate between multiple processes.
Regarding the performance, one of my colleagues have performed experiments before using 4 Tesla V100 GPUs using the codes from here. The results suggested that 3 settings work the best: replicated with all_reduce_spec=nccl, collective_all_reduce with properly tuned allreduce_merge_scope (e.g. 32), and horovod. I did not see significant differences among these 3.
For example if i want to use renset_v2, there are two model file on tensorflow:
one is here, another is here. Lots of tensorflow model are both in models/research and tensorflow/contrib.
I am very confused: which model is better? which model should i use?
In general, tf.contrib contains contributed code mostly by the community. It is meant to contain features and contributions that eventually should get merged into core TensorFlow, but whose interfaces may still change, or which require some testing to see whether they can find broader acceptance.
The code in tf.contrib isn't supported by the Tensorflow team. It is included in the hope that it is helpful, but it might change or be removed at any time; there are no guarantees.
tf.research folder contains machine learning models implemented by researchers in TensorFlow. The models are maintained by their respective authors and have a lower chance of being deprecated.
On the other hand models present directly are officially supported by Tensorflow team and are generally preferred as they have a lower chance of being deprecated in future releases, If you have a model implemented in both, you should generally avoid using the contrib version keeping in mind future compatibility, but the community does do some awesome stuff there, so you might find some models/work not present in the main repository but would be helpful if you used them directly from contrib branch.
Also notice the phrase generally avoid since it is a bit application dependent.
Hope that answers your question, comment with your doubts.
With Tensorflow 2.0 (that will come soon) tf.contrib will be removed.
Therefore, you have to start using models/research if you want your project to be up-to-date and still working in the next months.
Following this answer I wonder what are the principal differences (if any) between WriteToBigQuery and BigQuerySink of the Apache Beam Python SDK.
What are the considerations or limitations of using one over another?
Looking at sources:
BigQuerySink triggers a Dataflow native sink for BigQuery
that only supports batch pipelines. Instead of using this sink
directly, please use WriteToBigQuery transform that works for both
batch and streaming pipelines.
They both seem to do a similar thing underneath otherwise.