I am trying to use tensorflow-probability and started off with something really simple:
import tensorflow as tf
import tensorflow_probability as tfp
tf.enable_eager_execution()
tfd = tfp.distributions
poiss = tfd.Poisson(0.8)
poiss.sample(2, seed=1)
#> Out: <tf.Tensor: id=3569, shape=(2,), dtype=float32, numpy=array([0., 0.], dtype=float32)>
poiss.sample(2, seed=1)
#> Out: <tf.Tensor: id=3695, shape=(2,), dtype=float32, numpy=array([1., 0.], dtype=float32)>
poiss.sample(2, seed=1)
#> Out: <tf.Tensor: id=3824, shape=(2,), dtype=float32, numpy=array([2., 2.], dtype=float32)>
poiss.sample(2, seed=1)
#> Out: <tf.Tensor: id=3956, shape=(2,), dtype=float32, numpy=array([0., 1.], dtype=float32)>
I was thinking I would get the same results when re-using the same seed, but somehow that's not true.
I also tried without eager execution, but the results still weren't reproducible. Same story if I add something like tf.set_random_seed(12).
I suppose there is something basic I am missing?
For those interested, I am running Python 3.5.2 on Ubuntu 16.04 with
tensorflow-probability==0.5.0
tensorflow==1.12.0
For deterministic output in graph mode, you need to set both the graph random seed (tf.set_random_seed) and the op random seed (seed= in your sample call).
The workings of random samplers in TFv2 are still being sorted out. For now, my best understanding is that you can call tf.set_random_seed prior to each call to a sampler, and pass the sampler a seed=, if you want deterministic output in eager.
This is now cleaner, we support fully deterministic randomness in TFP. You can pass a tuple of two ints for seed, or a Tensor of shape (2,) to trigger the deterministic behavior. tfp.random.split_seed is also relevant here.
Besides setting the seed for sample or sample_chain in mcmc, you might need to set the followings as well:
seed = 24
os.environ['TF_DETERMINISTIC_OPS'] = 'true'
os.environ['PYTHONHASHSEED'] = f'{seed}'
np.random.seed(seed)
random.seed(seed)
tf.random.set_seed(seed)
Related
I have to build a model in keras. I am really struggling with my actual dataset, hence I am just trying to figure out the basics on a simpler dataset.
model = Sequential([
Dense(32, input_dim=X_train.shape[1], activation="sigmoid"),
Dense(2, activation="softmax"),
])
metrics=[
tf.keras.metrics.TruePositives(name="tp"),
tf.keras.metrics.TrueNegatives(name="tn"),
tf.keras.metrics.FalseNegatives(name="fn"),
tf.keras.metrics.FalsePositives(name="fp"),
tf.keras.metrics.Recall(name="recall"),
tf.keras.metrics.Precision(name="precision")
]
model.compile(loss="categorical_crossentropy", metrics=metrics, optimizer="sgd")
model.evaluate(X_test, y_test)
evaluation = model.evaluate(X_test, y_test)
for i, m in model.metrics_names:
print(m, evaluation[i])
This gets printed out:
loss 0.4604386021425058
tp 2965.5
tn 2965.5
fn 531.25
fp 531.25
recall 0.8480753898620605
precision 0.8480753898620605
Something really strange about this results. I believe it is due to using the softmax with two nodes.
y_train looks something like this:
array([[1., 0.],
[1., 0.],
[1., 0.],
[1., 0.]], dtype=float32)
I tried a sigmoid, but then the whole model breaks down, at least here the fitting works.
Is there a way to configure recall and precision so they consider one output notes as Positive?
The only solution in your case is to transform the problem into a one-dimensional one, i.e.
Use Dense(1,activation='sigmoid') instead of Dense(2,activation='softmax'); change [0,1] to 0 and [1,0] to 1 as an example.
Use binary_crossentropy instead of categorical_crossentropy.
Otherwise, you can implement a special callback to retrieve those metrics (using scikit-learn, like in the example below):
How to get other metrics in Tensorflow 2.0 (not only accuracy)?
I'm a beginner at TensorFlow, and my book said I should put this code first to produce the same sequence of results.
seed = 3
np.random.seed(seed)
tf.random.set_seed(seed)
I put various values in seed, and the results showed a big difference.
Does it affect the initial value setting of weights?
What exactly does that seed value setting play?
This is not a term which is specific for TensorFlow. Almost every programming language have a seed for to determine randomness of the random number generators etc. With seeds you can re-produce your results. For to reproduce results you need to restart kernel see the difference. For example:
import numpy as np
import tensorflow as tf
from tensorflow.keras.layers import Dense
from tensorflow.keras.models import Sequential
seed = 3
np.random.seed(seed)
tf.random.set_seed(seed)
model = Sequential()
model.add(Dense(2, activation="relu", input_shape = (28,28,1), kernel_initializer=initializer))
model.add(Dense(1, kernel_initializer=initializer))
model.compile(optimizer="adam", loss="sparse_categorical_crossentropy", metrics=['accuracy'])
This will give, 1st run:
>> model.get_weights()
[array([[-1.036157 , 0.19191754]], dtype=float32),
array([0., 0.], dtype=float32),
array([[-0.86388123],
[-0.8873284 ]], dtype=float32),
array([0.], dtype=float32)]
If you create the same model for the second time:
>> model.get_weights()
[array([[-0.4151404 , 0.81533253]], dtype=float32),
array([0., 0.], dtype=float32),
array([[ 0.39391458],
[-0.48291653]], dtype=float32),
array([0.], dtype=float32)]
produce the same sequence of results
Now, restart the kernel and create the model again. Results should be same for first and second run.
If you want to create a model with exact same weights everytime, you need to do:
initializer = tf.keras.initializers.GlorotNormal(seed=seed)
model.add(Dense(2, activation="relu", input_shape = (28,28,1), kernel_initializer=initializer))
model.add(Dense(1, kernel_initializer=initializer))
>> model.get_weights()
[array([[-0.6233864, 1.5557635]], dtype=float32),
array([0., 0.], dtype=float32),
array([[-0.6233864],
[ 1.5557635]], dtype=float32),
array([0.], dtype=float32)]
will be same, does not matter how many times you create the model, even after restarting kernel.
I am trying to port the implementation of concrete dropout in keras in https://github.com/yaringal/ConcreteDropout/blob/master/concrete-dropout-keras.ipynb to tensorflow 2. This is mostly straightforward, as tf 2 has most of the keras API built into it. However, the custom losses are being cleared before fitting.
After the model is defined, and before compiling it, I can see that the losses for each concrete dropout layer have been added to the model losses by the line self.layer.add_loss(regularizer) run when the layers are built:
>>> print(model.losses)
[<tf.Tensor: id=64, shape=(), dtype=float32, numpy=-8.4521576e-05>, <tf.Tensor: id=168, shape=(), dtype=float32, numpy=-0.000650166>, <tf.Tensor: id=272, shape=(), dtype=float32, numpy=-0.000650166>, <tf.Tensor: id=376, shape=(), dtype=float32, numpy=-0.000650166>, <tf.Tensor: id=479, shape=(), dtype=float32, numpy=-0.000650166>]
After the compilation, however, model.losses becomes an empty list, and the assertion assert len(model.losses) == 5 fails. If I choose to ignore the assertion, the fact that the layer losses are being neglected shows up in the warning WARNING:tensorflow:Gradients do not exist for variables ['concrete_dropout/p_logit:0', 'concrete_dropout_1/p_logit:0', 'concrete_dropout_2/p_logit:0', 'concrete_dropout_3/p_logit:0', 'concrete_dropout_4/p_logit:0'] when minimizing the loss. when training the model.
After digging into the compilation code in https://github.com/tensorflow/tensorflow/blob/r2.0/tensorflow/python/keras/engine/training.py#L184 I believe the problematic lines are
# Clear any `_eager_losses` that was added.
self._clear_losses()
Why is this being done at model compilation? And how can I add a loss per layer in tensorflow 2, if this is not the way to do it?
Since that custom loss is not dependent on the model's inputs, you should add it using a zero-argument callable like this:
self.layer.add_loss(lambda: regularizer)
My code has similar pattern with tensorflow 2.0 tutorial.
I want my dataset object to reshuffle in every epochs.
dataset = tf.data.Dataset.from_tensor_slices(['a','b','c','d'])
dataset = dataset.shuffle(100)
for epoch in range(10):
for d in dataset:
print(d)
Result:
tf.Tensor(b'c', shape=(), dtype=string)
tf.Tensor(b'a', shape=(), dtype=string)
tf.Tensor(b'b', shape=(), dtype=string)
tf.Tensor(b'd', shape=(), dtype=string)
tf.Tensor(b'c', shape=(), dtype=string)
tf.Tensor(b'a', shape=(), dtype=string)
tf.Tensor(b'b', shape=(), dtype=string)
tf.Tensor(b'd', shape=(), dtype=string)
...
It seems the dataset doesn't shuffle for each epoch.
Should I call .shuffle() for each epoch?
Yes, you should call .shuffle during the inner loop. Moreover, it is better to do not mix python code and TensorFlow code when pure tf.* method equivalent to the Python statements are available.
import tensorflow as tf
dataset = tf.data.Dataset.from_tensor_slices(["a", "b", "c", "d"])
# dataset = dataset.shuffle(2)
#tf.function
def loop():
for epoch in tf.range(10):
for d in dataset.shuffle(2):
tf.print(d)
loop()
The loop call produces the different values every time (and tf.print prints the content of the tf.Tensor, differently from print that prints the object).
What's the difference between tf.random.normal and tf.distributions.Normal? Or the difference between tf.distributions.Multinomial and tf.random.multinomial or anything similar?
Is tf.distributions.Normal used as the backend for tf.random.normal?
I recently looked at tf probability, the new place for tf distributions. This is my understanding:
They are not the same. tf.distributions.Normal will give you a distribution object from which you can sample (this will be same as evaluating the tensor returned by tf.random.normal function call for the same mean and loc values). But, a distribution additionally allows you to evaluate probability of a sample that you provide and all the aspects of having access to a distribution.
For example, you could do the following:
>>> import tensorflow as tf
>>> dist = tf.distributions.Normal(loc=0., scale=1.)
>>> dist.log_prob(tf.random.normal(shape=(3,3)))
<tf.Tensor: id=58, shape=(3, 3), dtype=float32, numpy=
array([[-0.9486696 , -0.95645994, -1.1610177 ],
[-1.244764 , -1.416851 , -1.1236244 ],
[-0.9292835 , -0.98901427, -0.9705758 ]], dtype=float32)>