Why is DRF (Distributed Random Forest) slower than XGBoost in h2o automl - xgboost

I am using H2O Auto ML and while comparing the time for XGBoost and D-RandomForest, DRF is taking a lot more time than XGBoost!
Code for DRF:
x = hf.columns
y = target_metric
x.remove(y)
aml = H2OAutoML(max_models=1, seed=1, include_algos=["DRF"])
aml.train(x=x, y=y, training_frame=hf)
m = aml.leader
varimp = m.varimp(use_pandas=True)
Code for XGboost:
x = hf.columns
y = target_metric
x.remove(y)
aml = H2OAutoML(max_models=1, seed=1, include_algos=["XGBoost"])
aml.train(x=x, y=y, training_frame=hf)
m = aml.leader
varimp = m.varimp(use_pandas=True)
H2O-AUTOML

Related

My model gives terrible results when im trying to forecast univariant time series

I am trying to do univariant time series forecasting. My model works Perfectly in different datasets. But for this dataset, the prediction is incredibly bad (tried 50,100,200 epochs, different batch sizes, and learning rates. Nothing has changed. So I think there is something wrong with my dataset.)
Here values of my dataset:
Mean: 49.840000, standard deviation: 31.786387
Here is my architecture
Here is the sample from my dataset
Here is my prediction values
Here is the code for normalization that im using:
def NormalizeMult(data):
#normalize 用于反归一化
data = np.array(data)
normalize = np.arange(2*data.shape[1],dtype='float64')
normalize = normalize.reshape(data.shape[1],2)
print(normalize.shape)
for i in range(0,data.shape[1]):
#第i列
list = data[:,i]
listlow,listhigh = np.percentile(list, [0, 100])
# print(i)
normalize[i,0] = listlow
normalize[i,1] = listhigh
delta = listhigh - listlow
if delta != 0:
#第j行
for j in range(0,data.shape[0]):
data[j,i] = (data[j,i] - listlow)/delta
#np.save("./normalize.npy",normalize)
return data,normalize
Here is the code that I'm using dataset and normalizing it:
INPUT_DIMS = 1
TIME_STEPS = 4
lstm_units = 64
#归一化
series = read_csv('/content/logs.csv')
series = series.drop(["timestamp"],axis=1)
series= series.dropna()
series = series.head(100)
data=series
data,normalize = NormalizeMult(data[0:50])
pollution_data = data[:,0].reshape(len(data[0:50]),1)
train_X, _ = split_sequence(data,TIME_STEPS)
_ , train_Y = split_sequence(data,TIME_STEPS)
optimizer = tf.keras.optimizers.Adam(lr=0.001)
m = attention_model()
m.summary()
m.compile(optimizer, loss='mse')
m.fit(train_X, train_Y, epochs=500, batch_size=2, validation_split=0.1)

Get FLOPS using GPflow

Is it somehow possible in GPflow to get FLOPS? I have found an example using Tensorflow but have no idea how to use this in a GPflow context:
g = tf.Graph()
run_meta = tf.RunMetadata()
with g.as_default():
A = tf.Variable(tf.random_normal( [25,16] ))
B = tf.Variable(tf.random_normal( [16,9] ))
C = tf.matmul(A,B)
opts = tf.profiler.ProfileOptionBuilder.float_operation()
flops = tf.profiler.profile(g, run_meta=run_meta, cmd='op', options=opts)
if flops is not None:
print('TF stats gives',flops.total_float_ops)
I dig a little bit in the source code of GPFlow. The key to make it work is to intercept the Tensorflow op that you want to profile before GPFlow's AutoFlows creates a new graph.
In my case, I wanted to profile the predict() function. The function you need is model._build_predict() (there's one equivalent for log-likelihood).
Here's how it works:
gpflow.reset_default_graph_and_session()
kernel = gpflow.kernels.RBF(1)
model = gpflow.models.GPR(X, Y, kernel)
run_metadata = tf.RunMetadata()
with model.enquire_session(session=None) as tf_session:
predict_op = model._build_predict(X)
tf_session.run(predict_op, options=tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE),
run_metadata=run_metadata)
opts = tf.profiler.ProfileOptionBuilder.float_operation()
prof = tf.profiler.profile(tf_session.graph, run_meta=run_metadata,
cmd='op', options=opts)
print('FOps: ', prof.total_float_ops)

Keras GAN (generator) not training well despite accurate discriminator

I've tried sorting this out for a few days now, following many pieces of advice found on forums etc, and now would welcome any suggestions to what is wrong!
I'm attempting to get my first GAN training - a simple feedforward deep net - very similar to using MNIST dataset, but with spectrum power windows derived from the VCTK-Corpus (size(1, 513)).
You can see from the Tensorboard graphs below that the networks seem to be interacting, and there is some sort of training going on:
Tensorboard graph overview. Tensorboard graph zoom.
However, results are poor and noisy: generated and validation comparison
The generator takes normal noise (usually 30 to 100 vectors) with a mean of 0 and stdev of 0.5.
def gan_generator(x_shape, frame_size):
g_input = Input(shape=x_shape)
H = BatchNormalization()(g_input)
H = Dense(128)(H)
H = LeakyReLU()(H)
H = BatchNormalization()(H)
H = Dense(128)(H)
H = LeakyReLU()(H)
H = BatchNormalization()(H)
H = Dense(256)(H)
H = LeakyReLU()(H)
H = BatchNormalization()(H)
H = Dense(256)(H)
H = LeakyReLU()(H)
H = BatchNormalization()(H)
H = Dense(256)(H)
H = LeakyReLU()(H)
H = BatchNormalization()(H)
out = Dense(frame_size[1], activation='linear')(H)
generator = Model(g_input, out)
generator.summary()
return generator
The discriminator determines a one-hot categorisation of generated frames:
(not sure about batch normalisation here - I've read it shouldn't be used if you're mixing real and generated into one batch. However, the generator makes much more convincing results with it than without - despite having a higher loss.)
def gan_discriminator(input_shape):
d_input = Input(shape=input_shape)
H = Dropout(0.1)(d_input)
H = Dense(256)(H)
H = Dropout(0.1)(H)
H = LeakyReLU()(H)
H = BatchNormalization()(H)
H = Dense(128)(H)
H = Dropout(0.1)(H)
H = LeakyReLU()(H)
H = BatchNormalization()(H)
H = Dense(100)(H)
H = Dropout(0.1)(H)
H = LeakyReLU()(H)
H = BatchNormalization()(H)
H = Dense(100)(H)
H = Dropout(0.1)(H)
H = LeakyReLU()(H)
H = BatchNormalization()(H)
H = Reshape((1, -1))(H)
d_V = Dense(2, activation='softmax')(H)
discriminator = Model(d_input,d_V)
discriminator.summary()
return discriminator
The GAN is easy:
def init_gan(generator, discriminator):
x = Input(shape=generator.inputs[0].shape[1:])
#Generator makes a prediction
pred = generator(x)
#Discriminator attempts to categorise prediction
y = discriminator(pred)
GAN = Model(x, y)
return GAN
Some training variables:
GAN (generator): Adam, lr=1e-4, categorical_crossentropy
discriminator: Adam, lr=1e-3, categorical_crossentropy
batch size: around 8000 samples
Mini-batch (weight update cycle): 32
The training loop:
#Pre-training Discriminator Network
#Load new batch of real frames
frames = load_data(data_dir)
frames_label = np.zeros((frames.shape[0], 1, 2))
frames_label[:, :, 0] = 1 #mark as real frames
#Generate Frames from noise vector
X_noise = noisegen((frames.shape[0], 1, n_noise))
generated_frames = generator.predict(X_noise)
generated_label = np.zeros((generated_frames.shape[0], 1, 2))
generated_label[:, :, 1] = 1 #mark as false frames
#Prep Data - concat real and false data
dis_batch_x = np.concatenate((frames, generated_frames), axis=0)
dis_batch_y = np.concatenate((frames_label, generated_label), axis=0)
#Make discriminator trainable and train for 8 epochs
make_trainable(discriminator, True)
discriminator.compile(optimizer=dis_optimizer, loss=dis_loss)
fit_model(discriminator, dis_batch_x, dis_batch_y, 8)
#Training Loop
for d in range(data_sets):
print "Starting New Dataset: {0}/{1}".format(d+1, data_sets)
""" Fit Discriminator """
#Load new batch of real frames
frames = load_data(data_dir)
frames_label = np.zeros((frames.shape[0], 1, 2))
frames_label[:, :, 0] = 1 #mark as real frames
#Generate Frames from noise vector
X_noise = noisegen((frames.shape[0], 1, n_noise))
generated_frames = generator.predict(X_noise)
generated_label = np.zeros((generated_frames.shape[0], 1, 2))
generated_label[:, :, 1] = 1 #mark as false frames
#Prep Data - concat real and false data
dis_batch_x = np.concatenate((frames, generated_frames), axis=0)
dis_batch_y = np.concatenate((frames_label, generated_label), axis=0)
#Make discriminator trainable & fit
make_trainable(discriminator, True)
discriminator.compile(optimizer=dis_optimizer, loss=dis_loss)
fit_model(discriminator, dis_batch_x, dis_batch_y)
""" Fit Generator """
#Prep Data
X_noise = noisegen((frames.shape[0], 1, n_noise))
generated_label = np.zeros((generated_frames.shape[0], 1, 2))
generated_label[:, :, 1] = 1 #mark as false frames
make_trainable(discriminator, False)
GAN.layers[2].trainable = False #done twice just to be sure
GAN.compile(optimizer=GAN_optimizer, loss=GAN_loss)
fit_model(GAN, X_noise, generated_label)
And finally a little bit of system info:
OSX 10.12
Tensorflow 1.5.0 (GPU)
Keras 2.1.3
Python 2.7
Many thanks in advance!
What the solution actually was that I didn't swap my True/False class in Generator training (suggested https://github.com/soumith/ganhacks), which I think effectively makes it gradient ascent.
Clarification on this would be nice to have.

If I don't want to train in batches and my state is a vector, what should my tensors have for a shape?

I'm trying to use tensorflow to solve a reinforced learning problem. I created an gym environment of my own. The state is a one dimensional array (size 224) and there are 170 actions to choose from (0...169). I do not want to train in batches. What I want is to make the most simple version of the RL problem running with tensorflow.
My main problem is, i guess the dimensions. I would assume that TF would allow me to input the state as 1D tensor. But then I get an error when I want to calculate W*input=action. Dimensions error make it hard to know whats right. Also, examples on the web focus on training from images, in batches.
In general, I started in this tutorial, but the state is encoded differently, which again makes it hard to follow (especially since I'm not really familiar with python).
import gym
import numpy as np
import random
import tensorflow as tf
env = gym.make('MyOwnEnv-v0')
n_state = 224
n_action = 170
sess = tf.InteractiveSession()
# Implementing the network itself
inputs1 = tf.placeholder(shape=[1,n_state],dtype=tf.float32)
W = tf.Variable(tf.random_uniform([n_state,n_action],0,0.01))
Qout = tf.transpose(tf.matmul(inputs1,W))
predict = tf.reshape(tf.argmax(Qout,1), [n_action,1])
#Below we obtain the loss by taking the sum of squares difference between the target and prediction Q values.
nextQ = tf.placeholder(shape=[n_action,1],dtype=tf.float32)
loss = tf.reduce_sum(tf.square(nextQ - Qout))
trainer = tf.train.GradientDescentOptimizer(learning_rate=0.1)
updateModel = trainer.minimize(loss)
# Training the network
init = tf.global_variables_initializer()
print("input: ", inputs1.get_shape()
, "\nW: ", W.get_shape()
, "\nQout: ", Qout.get_shape()
, "\npredict:", predict.get_shape()
, "\nnextQ: ", nextQ.get_shape()
, "\nloss: ", loss.get_shape())
# Set learning parameters
y = .99
e = 0.1
num_episodes = 2000
#create lists to contain total rewards and steps per episode
jList = []
rList = []
with tf.Session() as sess:
sess.run(init)
for i in range(num_episodes):
#Reset environment and get first new observation
s = env.reset()
rAll = 0
d = False
j = 0
#The Q-Network
while j < 99:
j+=1
#Choose an action by greedily (with e chance of random action) from the Q-network
a,allQ = sess.run([predict,Qout],feed_dict={inputs1:s})
if np.random.rand(1) < e:
a = env.action_space.sample()
#Get new state and reward from environment
s1,r,d,_ = env.step(a)
#Obtain the Q' values by feeding the new state through our network
Q1 = sess.run(Qout,feed_dict={inputs1:s1})
#Obtain maxQ' and set our target value for chosen action.
maxQ1 = np.max(Q1)
targetQ = allQ
#targetQ[0,a[0]] = r + y*maxQ1
targetQ[a,0] = r + y*maxQ1
#Train our network using target and predicted Q values
_,W1 = sess.run([updateModel,W],feed_dict={inputs1:s,nextQ:targetQ})
rAll += r
s = s1
if d == True:
#Reduce chance of random action as we train the model.
e = 1./((i/50) + 10)
break
jList.append(j)
rList.append(rAll)
print('Percent of succesful episodes: ' + str(sum(rList)/num_episodes) + '%')

KeyError in Tensorflow when calling predict on trained model

I have trained a LinearRegressor with two features: x,y and the label: l
def train_input_fn():
x = [1,2,3,4]
y = [2,3,4,5]
feature_cols = tf.constant(x)
labels = tf.constant(y)
return feature_cols, labels
x = tf.contrib.layers.real_valued_column("x")
y = tf.contrib.layers.real_valued_column("y")
m = tf.contrib.learn.LinearRegressor(feature_columns=[ x,y],
model_dir=model_dir)
m.fit(input_fn=train_input_fn, steps=100)
After training I want to predict from two new values
new_sample = np.array([20,20])
m.predict(new_sample)
but I get this error message when calling predict
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/layers/python/layers/feature_column.py", line 870, in insert_transformed_feature
input_tensor = columns_to_tensors[self.name]
KeyError: 'x'
Does anyone know why I get KeyError?
Try this:
my_feature_columns = [tf.contrib.layers.real_valued_column("", dimension=2)]
m = tf.contrib.learn.LinearRegressor(feature_columns=my_feature_columns,
model_dir=model_dir)
m.fit(input_fn=train_input_fn, steps=100)
I am not an expert in Tensorflow but this works for me:
new_sample = np.array([20,20],dtype='float32')
empty_y = np.zeros(len(new_sample),dtype='float32')
prediction_x = tf.contrib.learn.io.numpy_input_fn({"x":new_sample},empty_y, batch_size=45, num_epochs=100)
forecast = list(estimator.predict(input_fn=prediction_x,as_iterable=False))