I would like to put some prior on the (hyper)parameters of the GP models in GPflow, but I cannot find any module (like gpflow.priors) or doc about this issue.
Besides, I noticed that prior is one of the arguments of the class parameter, together with a question.
GPflow 2.0 uses tensorflow_probability distributions objects for the priors, e.g.
model.kernel.lengthscales.prior = tensorflow_probability.distributions.Gamma(
gpflow.utilities.to_default_float(1.0), gpflow.utilities.to_default_float(1.0)
)
(or pass the distribution object as the prior argument of the gpflow.Parameter class). Note that TensorFlow by default uses float32, whereas with GPs we typically want to work in float64 - hence the calls to to_default_float.
This is mentioned in the documentation, both in the notebook on understanding models and discussed in depth in the notebook on how to use MCMC with GPflow.
Related
I'm trying to learn how to use XLA for my models. And I'm looking at the doc from official here: https://www.tensorflow.org/xla#enable_xla_for_tensorflow_models. It was documented that there are two methods to enable XLA: 1) Explicit compilation by using #tf.function(jit_compile=True) to decorate your training function. 2) Auto-clustering by setting environment variables.
As I'm using tensorflow 1.15, not 2.x. So I think the second approach is the same as using this statement:
config.graph_options.optimizer_options.global_jit_level = (
tf.OptimizerOptions.ON_1)
You can also found info from here: https://www.tensorflow.org/xla/tutorials/autoclustering_xla. It seems this is what they used in tf2.x:
tf.config.optimizer.set_jit(True) # Enable XLA.
I think they are the same, correct me if I'm wrong.
OK, so if using the first approach, I think in tf1.15, this is equivalent to using
tf.xla.experimental.compile(computation)
So, my question is if I have used
tf.xla.experimental.compile(computation) to decorate my whole training function. Is this equivalent to use
config.graph_options.optimizer_options.global_jit_level = (
tf.OptimizerOptions.ON_1)
? Anybody knows? Much appreciated.
According to this video from TF team (2021), clustering will automatically look for places to optimize. Nevertheless, due to an unpredictable behaviour, they recommend decorating tf.fuctions with #tf.function(jit_compile=True) over using out-of-the-box clustering.
In case you want to use autoclustering, set_jit(True) is being deprecated and the most correct way now is tf.config.optimizer.set_jit('autoclustering')
Does tensorflow-federated support assigning different hyper-parameters(like batch-size or learning rate) for different simulated devices?
Currently, you may find this a bit unnatural, but yes, such a thing is possible.
One approach to doing this that is supported today is to have each client take its local learning rate as a top-level parameter, and use this in the training. A dummy example here would be (sliding the model parameter in the computations below) something along the lines of
#tff.tf_computation(tff.SequenceTyoe(...), tf.float32)
def train_with_learning_rate(ds, lr):
# run training with `tf.data.Dataset` ds and learning rate lr
...
#tff.federated_computation(tff.FederatedType([tff.SequenceType(...), tf.float32])
def run_one_round(datasets_and_lrs):
return tff.federated_mean(
tff.federated_map(train_with_learning_rate, datasets_and_lrs))
Invoking the federated computation here with a list of tuples with the first element of the tuple representing the clients data and the second element representing the particular client's learning rate, would give what you want.
Such a thing requires writing custom federated computations, and in particular likely defining your own IterativeProcess. A similar iterative process definition was recently open sourced here, link goes to the relevant local client function definition to allow for learning rate scheduling on the clients by taking an extra integer parameter representing the round number, it is likely a good place to look.
I am new to TensorFlow and just went through the eager execution tutorial and came across the tf.decode_csv function. Not knowing about it, I read the documentation. https://www.tensorflow.org/api_docs/python/tf/decode_csv
I don't really understand it.
The documentation says 'records: A Tensor of type string.'
So, my question is: What qualifies as a 'Tensor'?
I tried the following code:
dec_res = tf.decode_csv('0.1,0.2,0.3', [[0.0], [0.0], [0.0]])
print(dec_res, type(dec_res))
l = [[1,2,3],[4,5,6],[7,8,9]]
r = tf.reshape(l, [9,-1])
print(l, type(l))
print(r, type(r))
So the list dec_res contains tf.tensor objects. That seems reasonable to me. But is an ordinary string also a 'Tensor' according to the documentation?
Then I tried something else with the tf.reshape function. In the documentation https://www.tensorflow.org/api_docs/python/tf/reshape it says that 'tensor: A Tensor.' So, l is supposed to be a tensor. But it is not of type tf.tensor but simply a python list. This is confusing.
Then the documentation says
Returns:
A Tensor. Has the same type as tensor.
But the type of l is list where the type of r is tensorflow.python.framework.ops.Tensor. So the types are not the same.
Then I thought that TensorFlow is very generous with things being a tensor. So I tried:
class car(object):
def __init__(self, color):
self.color = color
red_car = car('red')
#test_reshape = tf.reshape(red_car, [1, -1])
print(red_car.color) # to check, that red_car exists.
Now, the line in comments results in an error.
So, can anyone help me to find out, what qualifies as a 'Tensor'?
P.S.: I tried to read the source code of tf.reshape as given in the documentation
Defined in tensorflow/python/ops/gen_array_ops.py.
But this file does not exist in the Github repo. Does anyone know how to read it?
https://www.tensorflow.org/programmers_guide/tensors
TensorFlow, as the name indicates, is a framework to define and run
computations involving tensors. A tensor is a generalization of
vectors and matrices to potentially higher dimensions. Internally,
TensorFlow represents tensors as n-dimensional arrays of base
datatypes.
What you are observing commes from the fact that tensorflow operations (like reshape) can be built from various python types using the function tf.convert_to_tensor:
https://www.tensorflow.org/api_docs/python/tf/convert_to_tensor
All standard Python op constructors apply this function to each of
their Tensor-valued inputs, which allows those ops to accept numpy
arrays, Python lists, and scalars in addition to Tensor objects
TensorFlow provides a tf.summary.tensor_summary() function that appears to be a multidimensional variant of tf.summary.scalar():
tf.summary.tensor_summary(name, tensor, summary_description=None, collections=None)
I thought it could be useful for summarizing inferred probabilities per class ... somewhat like
op_summary = tf.summary.tensor_summary('classes', some_tensor)
# ...
summary = sess.run(op_summary)
writer.add_summary(summary)
However it appears that TensorBoard doesn't provide a way to display these summaries at all. How are they meant to be used?
I cannot get it to work either. It seems like that feature is still under development. See this video from the TensorFlow Dev Summit that states that the tensor_summary is still under development (starting at 9:17): https://youtu.be/eBbEDRsCmv4?t=9m17s. It will probably be better defined and examples should be provided in the future.
Is there a way in TensorFlow to find out if two graphs have the same structure ?
I am designing an abstract class whose individual instances are expected to represent different architectures. I have provided an abc.abstractmethod get() which defines the graph. However, I also want to be able to load a pre-trained graph from disk. I want to check if the pre-trained graph has the same definition as the one mentioned in the get() method of a concrete class.
How may I achieve this structural comparison ?
You can get graph definition of current graph as str(tf.get_default_graph().as_graph_def()) and compare for exact equality against your previous result.
Also, TensorFlow tests have more advanced function EqualGraphDef which can tell that two graphs are equal even when graph format has changed, ie, if actual and expected as GraphDef proto objects, you could do
from tensorflow.python import pywrap_tensorflow
diff = pywrap_tensorflow.EqualGraphDefWrapper(actual.SerializeToString(),
expected.SerializeToString())
assert not diff