I'm trying to learn how to use XLA for my models. And I'm looking at the doc from official here: https://www.tensorflow.org/xla#enable_xla_for_tensorflow_models. It was documented that there are two methods to enable XLA: 1) Explicit compilation by using #tf.function(jit_compile=True) to decorate your training function. 2) Auto-clustering by setting environment variables.
As I'm using tensorflow 1.15, not 2.x. So I think the second approach is the same as using this statement:
config.graph_options.optimizer_options.global_jit_level = (
tf.OptimizerOptions.ON_1)
You can also found info from here: https://www.tensorflow.org/xla/tutorials/autoclustering_xla. It seems this is what they used in tf2.x:
tf.config.optimizer.set_jit(True) # Enable XLA.
I think they are the same, correct me if I'm wrong.
OK, so if using the first approach, I think in tf1.15, this is equivalent to using
tf.xla.experimental.compile(computation)
So, my question is if I have used
tf.xla.experimental.compile(computation) to decorate my whole training function. Is this equivalent to use
config.graph_options.optimizer_options.global_jit_level = (
tf.OptimizerOptions.ON_1)
? Anybody knows? Much appreciated.
According to this video from TF team (2021), clustering will automatically look for places to optimize. Nevertheless, due to an unpredictable behaviour, they recommend decorating tf.fuctions with #tf.function(jit_compile=True) over using out-of-the-box clustering.
In case you want to use autoclustering, set_jit(True) is being deprecated and the most correct way now is tf.config.optimizer.set_jit('autoclustering')
Related
I would like to put some prior on the (hyper)parameters of the GP models in GPflow, but I cannot find any module (like gpflow.priors) or doc about this issue.
Besides, I noticed that prior is one of the arguments of the class parameter, together with a question.
GPflow 2.0 uses tensorflow_probability distributions objects for the priors, e.g.
model.kernel.lengthscales.prior = tensorflow_probability.distributions.Gamma(
gpflow.utilities.to_default_float(1.0), gpflow.utilities.to_default_float(1.0)
)
(or pass the distribution object as the prior argument of the gpflow.Parameter class). Note that TensorFlow by default uses float32, whereas with GPs we typically want to work in float64 - hence the calls to to_default_float.
This is mentioned in the documentation, both in the notebook on understanding models and discussed in depth in the notebook on how to use MCMC with GPflow.
I successfully managed to implement learning to rank by following the tutorial TF-Ranking for sparse features using the ANTIQUE question answering dataset.
Now my goal is to successfully save the learned model to disk so that I can easily load it without training again. Due to the Tensorflow docs, the estimator.export_saved_model() method seems to be the way to go. But I can't wrap my head around how to tell Tensorflow how my feature structure looks like. Due to the docs here the easiest way seems to be calling tf.estimator.export.build_parsing_serving_input_receiver_fn(), which returns me the required inpur receiver function which I have to pass to the export_saved_model function. But how do I tell Tensorflow how my features from my learning to rank model look like?
From my current understanding the model has context feature specs and example feature specs. So I guess I somehow have to combine those two specs into one feature description, which I then can pass to the build_parsing_serving_input_receiver_fn function?
So I think you are on the right track;
You can get a build_ranking_serving_input_receiver_fn like this: (substitue context_feature_columns(...) and example_feature_columns(...) with defs you probably have for creating your own context and example structures for your training data):
def example_serving_input_fn():
context_feature_spec = tf.feature_column.make_parse_example_spec(
context_feature_columns(_VOCAB_PATHS).values())
example_feature_spec = tf.feature_column.make_parse_example_spec(
list(example_feature_columns(_VOCAB_PATHS).values()))
servingInputReceiver = tfr.data.build_ranking_serving_input_receiver_fn(
data_format=tfr.data.ELWC,
context_feature_spec=context_feature_spec,
example_feature_spec=example_feature_spec,
list_size=_LIST_SIZE,
receiver_name="input_ranking_data",
default_batch_size=None)
return servingInputReceiver
And then pass this to export_saved_model like this:
ranker.export_saved_model('path_to_save_model', example_serving_input_fn())
(ranker here is a tf.estimator.Estimator, maybe you called this 'estimator' in your code)
ranker = tf.estimator.Estimator(
model_fn=model_fn,
model_dir=_MODEL_DIR,
config=run_config)
Is there any function in Tensorflow which does the exact same thing as theano.tensor.switch(cond, ift, iff)?
In TensorFlow you can use tf.cond(https://www.tensorflow.org/api_docs/python/tf/cond). There are some examples in the documentation.
Edit: As you mentioned, this operation is not element wise, then it's equivalent is tf.where (https://www.tensorflow.org/api_docs/python/tf/where)
The tensorflow documentation does not provide any example of how to perform a periodic evaluation of the model on an evaluation set.
Some people suggested the use of an Experiment, which sounds great but unfortunately does not work (depreciation and triggers an error).
Others suggested the use of SummarySaverHook, but I don't see how you can use that with an evaluation set (as opposed to the training set).
A solution would be to do the following
for i in range(number_of_epoch):
estimator.train(...) // on training set
estimator.evaluate(...) // on evaluation set
This architecture is explicitly discouraged in this paper (page 4 top right).
Any other idea/implementation?
EDIT:
The error message when running the experiment is the following:
File ".../anaconda2/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 253, in train if (config.environment != run_config.Environment.LOCAL and
AttributeError: 'RunConfig' object has no attribute 'environment'
Tensorflow version 1.3
Only a few parameters/options of Experiment are deprecated (what specific errors are you seeing?). If you create an Estimator that will do periodic checkpoints (using options in RunConfig) and an Experiment using it, you will get evaluation for each checkpoint by default when using train_and_evaluate method.
EDIT: As Maxime pointed out in the comments. He needed to add the following lines to get rid of his error:
os.environ['TF_CONFIG'] = json.dumps({'environment': 'local'})
config = tf.contrib.learn.RunConfig()
TensorFlow provides a tf.summary.tensor_summary() function that appears to be a multidimensional variant of tf.summary.scalar():
tf.summary.tensor_summary(name, tensor, summary_description=None, collections=None)
I thought it could be useful for summarizing inferred probabilities per class ... somewhat like
op_summary = tf.summary.tensor_summary('classes', some_tensor)
# ...
summary = sess.run(op_summary)
writer.add_summary(summary)
However it appears that TensorBoard doesn't provide a way to display these summaries at all. How are they meant to be used?
I cannot get it to work either. It seems like that feature is still under development. See this video from the TensorFlow Dev Summit that states that the tensor_summary is still under development (starting at 9:17): https://youtu.be/eBbEDRsCmv4?t=9m17s. It will probably be better defined and examples should be provided in the future.