TensorFlow XOR implementation, fail to achieve 100% accuracy - tensorflow

Question

I am a newbie in machine learning and tensorflow. I am trying to implement XOR gate in tensor flow I have come up with this code.
import numpy as np
import tensorflow as tf
tf.reset_default_graph()
learning_rate = 0.01
n_epochs = 1000
n_inputs = 2
n_hidden1 = 2
n_outputs = 2
arr1, target = [[0, 0], [0, 1], [1, 0], [1,1]], [0, 1, 1, 0]
X_data = np.array(arr1).astype(np.float32)
y_data = np.array(target).astype(np.int)
X = tf.placeholder(tf.float32, shape=(None, n_inputs), name="X")
y = tf.placeholder(tf.int64, shape=(None), name="y")
with tf.name_scope("dnn_tf"):
hidden1 = tf.layers.dense(X, n_hidden1, name="hidden1", activation=tf.nn.relu)
logits = tf.layers.dense(hidden1, n_outputs, name="outputs")
with tf.name_scope("loss"):
xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)
loss = tf.reduce_mean(xentropy, name="loss")
with tf.name_scope("train"):
optimizer = tf.train.MomentumOptimizer(learning_rate, momentum=0.9)
training_op = optimizer.minimize(loss)
with tf.name_scope("eval"):
correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))
init = tf.global_variables_initializer()
with tf.Session() as sess:
init.run()
for epoch in range(n_epochs):
if epoch % 100 == 0:
print("Epoch: ", epoch, " Train Accuracy: ", acc_train)
sess.run(training_op, feed_dict={X:X_data, y:y_data})
acc_train = accuracy.eval(feed_dict={X:X_data, y:y_data})
The code runs fine but I am getting different outputs in each run
Run-1
Epoch: 0 Train Accuracy: 0.75
Epoch: 100 Train Accuracy: 1.0
Epoch: 200 Train Accuracy: 1.0
Epoch: 300 Train Accuracy: 1.0
Epoch: 400 Train Accuracy: 1.0
Epoch: 500 Train Accuracy: 1.0
Epoch: 600 Train Accuracy: 1.0
Epoch: 700 Train Accuracy: 1.0
Epoch: 800 Train Accuracy: 1.0
Epoch: 900 Train Accuracy: 1.0
Run -2
Epoch: 0 Train Accuracy: 1.0
Epoch: 100 Train Accuracy: 0.75
Epoch: 200 Train Accuracy: 0.75
Epoch: 300 Train Accuracy: 0.75
Epoch: 400 Train Accuracy: 0.75
Epoch: 500 Train Accuracy: 0.75
Epoch: 600 Train Accuracy: 0.75
Epoch: 700 Train Accuracy: 0.75
Epoch: 800 Train Accuracy: 0.75
Epoch: 900 Train Accuracy: 0.75
Run3-
Epoch: 0 Train Accuracy: 1.0
Epoch: 100 Train Accuracy: 0.5
Epoch: 200 Train Accuracy: 0.5
Epoch: 300 Train Accuracy: 0.5
Epoch: 400 Train Accuracy: 0.5
Epoch: 500 Train Accuracy: 0.5
Epoch: 600 Train Accuracy: 0.5
Epoch: 700 Train Accuracy: 0.5
Epoch: 800 Train Accuracy: 0.5
Epoch: 900 Train Accuracy: 0.5
I am unable to understand what I am doing wrong here and why my solution is not converging.

Answer 1

In theory it's possible to solve XOR with one hidden layer with two units with ReLU activations as you have in your code. However, there is always the crucial difference between a network being able to represent a solution and being able to learn it. I would assume that due to the small size of the network you run into the "dead ReLU" problem where due to unfortunate random initialization one (or both) of your hidden units doesn't activate for any input. Unfortunately ReLU also has zero gradient when it has zero activation, so a unit that never activates also cannot learn anything.
Increasing the number of hidden units makes it less likely that this happens (i.e. you can have three dead units and the other two will still be enough to solve the problem), which could explain why you are more successful with five hidden units.

Answer 2

You might want to check out the interactive TensorFlow Playground. They have a XOR dataset available. You can play around with the number of hidden layers, size, activation functions etc. and visualise the decision boundaries the classifier learns with the number of epohcs.