I am learning XLA recently. XLA (Accelerated Linear Algebra) is a domain-specific compiler for linear algebra that can accelerate TensorFlow models with potentially no source code changes. Due to the lack of document, it is hard to learn some components systematically.
I am trying to understand the matcher such as [m::MaximumAnyOrder][1], m::Broadcast.
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/compiler/xla/service/gpu/cudnn_fused_conv_rewriter.cc#L607-L608
But I didn't the the function definition or document on these APIs. It seems they are not in the codebase of XLA.
Any experts can help me how to find the API definition or document to better understand how they work?
Related
I'm trying to find a way to utilise Tensorflow Federated in C++. I know it's possible to do it for the regular Tensorflow with the Core API, however I can't find a way for Federated. If it's not possible suggestions for workarounds would be highly appreciated!
It would be helpful to know which part(s) of TFF you want to use in C++, and what your use case is, as that will influence the answer:
The APIs for defining federated computations (tff.federated_computation); as with TensorFlow, these are pretty tightly coupled to Python.
Executing serialized computations (stored as instances of computation.proto). This can conceptually be done using a purely C++ API, though TFF doesn't currently provide such a runtime.
TFF has since implemented a C++ runtime along the lines of Brendan's answer. TFF's CC directory contains the code; most of the implementation is in the executors directory.
These APIs can certainly be called from C++ code; see, e.g., the implementation of TFF's RunWorker, which starts and runs a server that can execute TFF computations.
I have been using TensorRT and TensorFlow-TRT to accelerate the inference of my DL algorithms.
Then I have heard of:
JAX https://github.com/google/jax
Trax https://github.com/google/trax
Both seem to accelerate DL. But I am having a hard time to understand them. Can anyone explain them in simple terms?
Trax is a deep learning framework created by Google and extensively used by the Google Brain team. It comes as an alternative to TensorFlow and PyTorch when it comes to implementing off-the-shelf state of the art deep learning models, for example Transformers, Bert etc. , in principle with respect to the Natural Language Processing field.
Trax is built upon TensorFlow and JAX. JAX is an enhanced and optimised version of Numpy. The important distinction about JAX and NumPy is that the former using a library called XLA (advanced linear algebra) which allows to run your NumPy code on GPU and TPU rather than on CPU like it happens in the plain NumPy, thus speeding up computation.
I'm really confused about the purpose of DeepMind's Sonnet library for TensorFlow. As far as I can tell from the documentation, it seems to do essentially what Keras does (flexible functional abstractions). Can someone tell me what the advantage of Sonnet is?
There isn't much difference between them. They are both:
High-level object oriented libraries that bring about abstraction when developing neural networks (NN) or other machine learning (ML) algorithms.
Built on top of TensorFlow (with the addition of Theano for Keras).
So why did they make Sonnet? It appears that Keras doesn't seem to suit the needs of DeepMind. So DeepMind came up with Sonnet, a high-level object oriented programming library built on top of TensorFlow to address its research needs.
Keras and Sonnet are both trying to simplify deep reinforcement learning, with the major difference being Sonnet is specifically adapted to the problems that DeepMind explores.
The main advantage of Sonnet, from my perspective, is you can use it to reproduce the research demonstrated in DeepMind's papers with greater ease than keras, since DeepMind will be using Sonnet themselves. Aside from that advantage, it's just yet another framework with which to explore deep RL problems.
Knowing that Tensorflow is good for working with matrices, would I be able to use Tensorflow to create a cellular automata? And would this offer a great deal of speed over just coding it in Python?
Are there any tutorials or websites that could point me in the right direction to use Tensorflow for more general purpose computing than machine learning (for example, simulations)?
If so, could someone help point me in the right direction to the type of Tensorflow commands I would need to learn to make this program? Thanks!
A TensorFlow implementation is likely to offer an improvement in execution time, especially if executed by GPU, since CA can be executed in parallel. See: https://cs.stackexchange.com/a/320/67726.
A starting point for TensorFlow in general might be the official guide and documentation, which do go beyond just machine learning. Also available are two tutorials on non-ML examples: Mandelbrot Set, Partial Differential Equations.
While TensorFlow is usually mentioned in the context of machine learning, it is worth noting that:
TensorFlow™ is an open source software library for high performance
numerical computation. Its flexible architecture allows easy
deployment of computation across a variety of platforms (CPUs, GPUs,
TPUs), and from desktops to clusters of servers to mobile and edge
devices.
Edit: here's an implementation and a tutorial about Conway's Game of Life using TF.
Why don't tutorials from TensorFlow use classes?
I.e. no OOP practices are used in all tutorials from TensorFlow.
Is it something conceptual? Is it matter of taste? Does something block the models from using OOP?
I would really appreciate if someone can give me at least basic explanation.
You can definitely use classes with Tensorflow (as this question shows).
As to why they're not used in the tutorials, I can only guess the choice was made to avoid adding another layer of complexity to understand the basic principles of Tensorflow. Understanding the graph, how TF variables relate to Python variables and TF scoping is hard enough without classes.
Nothing prevent you from using classes to implement your models. Have a look to this very nice post on how you could structure your TensorFlow models in an Object Oriented way in Python.