I am using the following code to see If I am able to stop TF/KERAS from producing logs.
import tensorflow as tf
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))
tf.debugging.set_log_device_placement(True)
# Create some tensors
a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])
b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])
c = tf.matmul(a, b)
print(c)
Here you may see that I have used os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' in my code to disable logs.
Num GPUs Available: 1 Executing op MatMul in device
/job:localhost/replica:0/task:0/device:GPU:0 tf.Tensor( [[22. 28.]
[49. 64.]], shape=(2, 2), dtype=float32)
Is there any way I can disable TF/KERAS to print Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0 ?
Remove the following line and you can get rid of the ops device placement messages:
tf.debugging.set_log_device_placement(True)
Related
I am training a tesorflow model on the GPU, but always get out of memory problem, so I want to set some of the operations to be run on CPU. Here is my code:
I set this in the main function:
gpus = tf.config.list_physical_devices(device_type='GPU')
tf.config.experimental.set_visible_devices(gpus[0],'GPU')
tf.config.experimental.set_memory_growth(gpus[0], enable=True)
And in the class(tf.keras.Model), for one of the function that needs to be called by init, I set
def _setup_C(self, double_length=False):
""" Construct C~ from C
double_length: current C is for length L, convert it to length 2L
"""
with tf.device('/cpu:0'):
C = _r2c(self.C)
self._setup_state()
dA_L = power(self.L, self.dA)
# Multiply C by I - dA_L
C_ = _conj(C)
prod = contract("h m n, c h n -> c h m", tf.transpose(dA_L,perm = [0,2,1]), C_)
if double_length: prod = -prod # Multiply by I + dA_L instead
C_ = C_ - prod
C_ = C_[..., :self.N] # Take conjugate pairs again
self.C = tf.identity(_c2r(C_))
if double_length:
self.L *= 2
self._omega(self.L, dtype=C.dtype, cache=True)
But it seems that it does not help with the problem, does anyone has some ideas for it?
You can use tf.device(),which specifies the device to be used for ops created/executed in a particular context. For example,i want to store the tensors on the CPU and perform a matrix multiplication operation on the GPU.
tf.debugging.set_log_device_placement(True)
# Place tensors on the CPU
with tf.device('/CPU:0'):
a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])
b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])
# Run on the GPU
c = tf.matmul(a, b)
print(c)
Output of the above code:
Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0
tf.Tensor(
[[22. 28.]
[49. 64.]], shape=(2, 2), dtype=float32)
Thank You.
I am trying to run a simple program on TPU:
import tensorflow as tf
tpu = tf.distribute.cluster_resolver.TPUClusterResolver()
print("Device:", tpu.master())
tf.config.experimental_connect_to_cluster(tpu)
tf.tpu.experimental.initialize_tpu_system(tpu)
strategy = tf.distribute.experimental.TPUStrategy(tpu)
a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])
b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])
with strategy.scope():
c = tf.matmul(a, b)
print("c device: ", c.device)
with tf.Session(config=tf.ConfigProto(log_device_placement=True)) as sess:
print(c.eval())
When I run this, it looks like the TPU is being found. However, none of the logged devices have 'TPU' in the name -- it is all on the CPU.
What am I doing wrong?
strategy.scope() is for model training.
If you want to run tf.matmul on a TPU you could use either this:
with tf.device('/TPU:0'):
c = tf.matmul(a, b)
Or
#tf.function
def matmul_fn(x, y):
z = tf.matmul(x, y)
return z
z = strategy.run(matmul_fn, args=(a, b))
print(z)
Details are here.
import tensorflow as tf
a=tf.random_normal([3, 2], mean=6, stddev=0.1, seed=1)
b=tf.random_normal([3, 2], mean=1, stddev=1, seed=1)
sess=tf.Session()
ra=sess.run(a)
rb=sess.run(b)
r1=ra-rb
r2=sess.run(tf.subtract(a,b))
Why is r1 and r2 not equal?
Shouldn't it be the same in theory？
tensorflow version : 1.15.0
In Tensorflow 1.x since in each session the tf.random_normal generates the new set of numbers which is the reason for change in results as rightly mentioned by #xdurch0 and #Addy in the comment section.
Instead, you can set the constant numbers using tf.constant and compare the results.
Tensorflow 1.x:
import tensorflow as tf
a = tf.constant([[5.918868 , 6.14846 ],
[6.006533 , 5.7557297],
[6.009925 , 6.0591226]])
b = tf.constant([[0.32409406, 1.2866583 ],
[1.3215888 , 2.2124639 ],
[0.19414288, 0.86650544]])
sess=tf.Session()
ra=sess.run(a)
rb=sess.run(b)
r1=ra -rb
r2=sess.run(tf.subtract(a,b))
print(r1)
print(r2)
Result:
[[5.5947742 4.8618016]
[4.684944 3.5432658]
[5.815782 5.192617 ]]
[[5.5947742 4.8618016]
[4.684944 3.5432658]
[5.815782 5.192617 ]]
Tensorflow 2.x:
In Tensorflow 2.x since eager execution is enabled by default the tf.random.normal will execute immediately and keep the result for rest of the code.
import tensorflow as tf
a=tf.random.normal([3, 2], mean=6, stddev=0.1, seed=1)
b=tf.random.normal([3, 2], mean=1, stddev=1, seed=1)
r1=a-b
r2=tf.subtract(a,b)
print(r1)
print(r2)
Result:
tf.Tensor(
[[5.5947742 4.8618016]
[4.684944 3.5432658]
[5.815782 5.192617 ]], shape=(3, 2), dtype=float32)
tf.Tensor(
[[5.5947742 4.8618016]
[4.684944 3.5432658]
[5.815782 5.192617 ]], shape=(3, 2), dtype=float32)
Here is the output of from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())
[name: "/device:CPU:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 8087623604945614369
]
Here is the output of pip list | grep tensorflow:
tensorflow-gpu (1.4.0)
tensorflow-tensorboard (0.4.0rc3)
I can confirm that I have installed cuda 8.0 and cudnn on my machine and the output of nvidia-smi shows the GPU along with other details. Can someone please help me to understand why the output from print(device_lib.list_local_devices()) doesn't show the GPU?rr
Tried this simple tensorflow example:
with tf.device('/gpu:0'):
a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
c = tf.matmul(a, b)
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
print(sess.run(c))
Error:
Operation was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0 ]
Link for the tf tutorial
# Creates a graph.
with tf.device('/cpu:0'):
a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
c = tf.matmul(a, b)
# Creates a session with log_device_placement set to True.
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
# Runs the op.
print sess.run(c)
In above example cpu:0 has assigned to the execution process. With the log_device_placement true. So this is solution for the above code that the have mentioned
Device mapping:
/job:localhost/replica:0/task:0/gpu:0 -> device: 0, name: Tesla K40c, pci bus
id: 0000:05:00.0
b: /job:localhost/replica:0/task:0/cpu:0
a: /job:localhost/replica:0/task:0/cpu:0
MatMul: /job:localhost/replica:0/task:0/gpu:0
[[ 22. 28.]
[ 49. 64.]]
Now here place holders a, b and the matmul operation c which runs inside a session is inside the device log cpu:o but in the log device description why only MatMul has been executed in gpu:0 ?
That seems like a bug in the documentation, the MatMul operation will be placed on CPU in this case.
Indeed, running the code sample does show this:
import tensorflow as tf
# Creates a graph.
with tf.device('/cpu:0'):
a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
c = tf.matmul(a, b)
# Creates a session with log_device_placement set to True.
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
# Runs the op.
print sess.run(c)
# And to prove that GPUs exist and can be used
with tf.device('/gpu:0'):
d = tf.random_normal([])
print sess.run(d)
Will show the following placements:
MatMul: (MatMul): /job:localhost/replica:0/task:0/cpu:0
b: (Const): /job:localhost/replica:0/task:0/cpu:0
a: (Const): /job:localhost/replica:0/task:0/cpu:0
random_normal/RandomStandardNormal: (RandomStandardNormal): /job:localhost/replica:0/task:0/gpu:0
I think the documentation bug is that the c = tf.matmul(a, b) statement was supposed to be outside the with tf.device('/cpu:0') scope.