Following two tower model tutorial , I made my recommendation model. Then I trained it for one epoch and evaluated it:
cached_train = train.shuffle(100_000).batch(4096)
cached_test = test.batch(4096).cache()
model = Model( layer_sizes = [ 32 ], use_context=False)
model.compile(optimizer=tf.keras.optimizers.Adagrad(0.001))
history = model.fit(cached_train, validation_data = cached_test, validation_freq=1,
callbacks=[tensorboard_callback],
epochs=1)
this model has following log (val_total_loss is less than training loss) :
684/684 [==============================] - 195s 284ms/step
- factorized_top_k/top_100_categorical_accuracy: 0.1404 - loss: 33958.1878 - regularization_loss: 0.0000e+00 - total_loss: 33958.1878 - val_factorized_top_k/top_100_categorical_accuracy: 0.1185 - val_loss: 25582.6953 - val_regularization_loss: 0.0000e+00 - **val_total_loss: 25582.6953**
Then I evaluated accuracy by model.evaluate :
test_accuracy = model.evaluate(
cached_test, return_dict=True)
171/171 [==============================] - 34s 197ms/step -
- factorized_top_k/top_100_categorical_accuracy: 0.1185 - loss: 33966.4893 - regularization_loss: 0.0000e+00 - **total_loss: 33966.4893**
Even though I didn't change model after training one epoch, total_loss in model.evaluate(cached_test) is different from the value I got by model.fit.
Why is this happening ?
Related
Is there any way to show the training progress from the TensorFlow linear estimator: tf.estimator.LinearClassifier().train() similar to how the progress output would be with a model.fit() for each Epoch?
tensorflow==2.9.2
Epoch 1/10
1875/1875 [==============================] - 5s 3ms/step - loss: 0.4964 - accuracy: 0.8270
Epoch 2/10
1875/1875 [==============================] - 4s 2ms/step - loss: 0.3751 - accuracy: 0.8652
Epoch 3/10
1875/1875 [==============================] - 5s 3ms/step - loss: 0.3382 - accuracy: 0.8762
Here is a sample of my code
#input function
def make_input_fn(data_df, label_df, num_epochs=1000, shuffle=True, batch_size=32):
def input_function(): # inner function, this will be returned
ds = tf.data.Dataset.from_tensor_slices((dict(data_df), label_df)) # create tf.data.Dataset object with data and its label
if shuffle:
ds = ds.shuffle(1000) # randomize order of data
ds = ds.batch(batch_size).repeat(num_epochs) # split dataset into batches of 32 and repeat process for number of epochs
return ds # return a batch of the dataset
return input_function # return a function object for use
train_input_fn = make_input_fn(dftrain, y_train) # here we will call the input_function that was returned to us to get a dataset object we can feed to the model
eval_input_fn = make_input_fn(dfeval, y_eval, num_epochs=1, shuffle=False)
pre_input_fn = make_input_fn(dfpre, y_pre, num_epochs=1, shuffle=False)
linear_est = tf.estimator.LinearClassifier(feature_columns=feature_columns)
linear_est.train(train_input_fn) # train
result = linear_est.evaluate(eval_input_fn)
What I've been doing, and I'm sure this isn't recommended (but I've not seen another method) is to run linear_est.train multiple times, and access the return of linear_est.evaluate() as so:
loss_by_epoch = list()
current_loss = 1.0
acceptable_loss = 0.4
while current_loss > acceptable_loss:
linear_est.train(train_input_fn)
result = linear_est.evaluate(eval_input_fn)
current_loss = result['loss']
loss_by_epoch.append(current_loss)
print(loss_by_epoch)
P.S. If anyone else wants to answer this question, feel free; this answer seems like the only way, and I hope it isn't.
I have training a deep neural network for classification task on my machine learning dataset.
On train as well as test set below are the observations:
For every true positive there are approx 3 false positive.
For approx 4 true negatives there is 1 false negatives
The data is scaled with standardscaler and then clipped between -5 and 5.
Below are observations while training.
382/382 [==============================] - 3s 9ms/step - loss: 0.6897 - tp: 84096.0000 - fp: 244779.0000 - tn: 355888.0000 - fn: 97448.0000 - accuracy: 0.5625 - precision: 0.2557 - recall: 0.4632 - auc: 0.5407 - prc: 0.2722
val_loss: 0.6838 - val_tp: 19065.0000 - val_fp: 56533.0000 - val_tn: 91902.0000 - val_fn: 23829.0000 - val_accuracy: 0.5800 - val_precision: 0.2522 - val_recall: 0.4445 - val_auc: 0.5468 - val_prc: 0.2722
Can someone expert please help to let me know what can I do to minimise false classification on train as well as test set.
I am using imbalanced dataset with class_weight as shown in below code:-
METRICS = [
keras.metrics.TruePositives(name='tp'),
keras.metrics.FalsePositives(name='fp'),
keras.metrics.TrueNegatives(name='tn'),
keras.metrics.FalseNegatives(name='fn'),
keras.metrics.BinaryAccuracy(name='accuracy'),
keras.metrics.Precision(name='precision'),
keras.metrics.Recall(name='recall'),
keras.metrics.AUC(name='auc'),
keras.metrics.AUC(name='prc', curve='PR'), # precision-recall curve
]
pos = sum(y_train)
neg = y_train.shape[0] - pos
total = y_train.shape[0]
weight_for_0 = (1 / neg) * (total / 2.0)
weight_for_1 = (1 / pos) * (total / 2.0)
class_weight = {0: weight_for_0, 1: weight_for_1}
def make_model(size, layers, metrics=METRICS, output_bias=None):
if output_bias is not None:
output_bias = tf.keras.initializers.Constant(output_bias)
model = keras.Sequential()
model.add(keras.layers.Dense(size,input_shape=(window_length*indicators,)))
model.add(keras.layers.Dropout(0.5))
for i in range(layers-1):
model.add(keras.layers.Dense(size))
model.add(keras.layers.Dropout(0.5))
model.add(keras.layers.Dense(1, activation = "sigmoid", bias_initializer=output_bias))
model.compile(
optimizer=keras.optimizers.Adam(learning_rate=0.0001),
loss=tf.keras.losses.BinaryCrossentropy(),
metrics=metrics)
return model
EPOCHS = 100
BATCH_SIZE = 2048
early_stopping = tf.keras.callbacks.EarlyStopping(
monitor='val_prc',
verbose=1,
patience=10,
mode='max',
restore_best_weights=True)
model = make_model(size = size,layers=layers, output_bias=np.log(pos/neg))
history = model.fit(X_train,y_train, batch_size=BATCH_SIZE, epochs=EPOCHS,callbacks=[early_stopping],validation_data=(X_test, y_test_pos)
,class_weight=class_weight
)
Can somebody please help
i am struggling with training a neural network that uses tf.data.DataSet as input.
What I find is that if I call .shuffle() before split the entire dataset in train, val, test set the accuracy on val (in training) and test (in evaluate) is 91%, but when I run .evaluate() on the test set many times the accuracy and loss metrics change every time. The same behavior occurs with .predict() on test set, with the classes that change every time.
This is the output of traning, evaluate end predict process
total_record: 93166 - trainin_size: 74534 - val_size: 9316 - test_size: 9316
Epoch 1/5
145/145 [==============================] - 42s 273ms/step - loss: 1.7143 - sparse_categorical_accuracy: 0.4051 - val_loss: 1.4997 - val_sparse_categorical_accuracy: 0.4885
Epoch 2/5
145/145 [==============================] - 40s 277ms/step - loss: 0.7571 - sparse_categorical_accuracy: 0.7505 - val_loss: 1.1634 - val_sparse_categorical_accuracy: 0.6050
Epoch 3/5
145/145 [==============================] - 41s 281ms/step - loss: 0.4894 - sparse_categorical_accuracy: 0.8223 - val_loss: 0.7628 - val_sparse_categorical_accuracy: 0.7444
Epoch 4/5
145/145 [==============================] - 38s 258ms/step - loss: 0.3417 - sparse_categorical_accuracy: 0.8656 - val_loss: 0.4236 - val_sparse_categorical_accuracy: 0.8579
Epoch 5/5
145/145 [==============================] - 40s 271ms/step - loss: 0.2660 - sparse_categorical_accuracy: 0.8926 - val_loss: 0.2807 - val_sparse_categorical_accuracy: 0.9105
accr = model.evaluate(test_set)
19/19 [==============================] - 1s 39ms/step - loss: 0.2622 - sparse_categorical_accuracy: 0.9153
accr = model.evaluate(test_set)
19/19 [==============================] - 1s 40ms/step - loss: 0.2649 - sparse_categorical_accuracy: 0.9170
accr = model.evaluate(test_set)
19/19 [==============================] - 1s 40ms/step - loss: 0.2726 - sparse_categorical_accuracy: 0.9141
accr = model.evaluate(test_set)
19/19 [==============================] - 1s 40ms/step - loss: 0.2692 - sparse_categorical_accuracy: 0.9166
pred = model.predict(test_set)
pred_class = np.argmax(pred, axis=1)
pred_class
Out[41]: array([0, 1, 5, ..., 2, 0, 1])
pred = model.predict(test_set)
pred_class = np.argmax(pred, axis=1)
pred_class
Out[42]: array([2, 3, 1, ..., 1, 2, 0])
pred = model.predict(test_set)
pred_class = np.argmax(pred, axis=1)
pred_class
Out[43]: array([1, 2, 4, ..., 1, 3, 0])
pred = model.predict(test_set)
pred_class = np.argmax(pred, axis=1)
pred_class
Out[44]: array([0, 3, 1, ..., 0, 5, 4])
So, I tried to apply .shuffle() after the split and only on the training and validation (commenting the main .shuffle() and uncommenting the shuffle in train_set and val_set).
But in this case, I find that the network goes into overfitting after just 5 epochs (with the previous training process callbacks block the training at 30° epochs with 94% val accuracy), with an accuracy of 75% since 2° epoch on validation set.
However, in this case if I run .evaluate() and .predict() on the testset to which .shuffle () has not been applied, the metrics and classes remain unchanged on each call.
Why this behavior?
But especially what is the great way and what is the real accuracy of model?
Thank's
This is the code of the process
""" ### Make tf.data.Dataset """
dataset = tf.data.Dataset.from_tensor_slices(({ "features_emb_subj": features_emb_subj,
"features_emb_snip": features_emb_snip,
"features_emb_fromcat": features_emb_fromcat,
"features_dense": features_dense,
"features_emb_user": features_emb_user}, cat_labels))
dataset = dataset.shuffle(int(len(features_dense)), reshuffle_each_iteration=True)
""" ### Split in train,val,test """
train_size = int(0.8 * len(features_dense))
val_size = int(0.10 * len(features_dense))
test_size = int(0.10 * len(features_dense))
test_set = dataset.take(test_size)
validation_set = dataset.skip(test_size).take(val_size)
training_set = dataset.skip(test_size + val_size)
test_set = test_set.batch(BATCH_SIZE, drop_remainder=False)
#validation_set = validation_set.shuffle(val_size, reshuffle_each_iteration=True)
validation_set = validation_set.batch(BATCH_SIZE, drop_remainder=False)
#training_set = training_set.shuffle(train_size, reshuffle_each_iteration=True)
training_set = training_set.batch(BATCH_SIZE, drop_remainder=True)
"""### Train model """
callbacks = [EarlyStopping(monitor='val_loss', patience=3, min_delta=0.0001, restore_best_weights=True)]
history = model.fit( training_set,
epochs = 5,
validation_data = validation_set,
callbacks=callbacks,
class_weight = setClassWeight(cat_labels),
verbose = 1)
"""### Evaluate model """
accr = model.evaluate(test_set)
"""### Predict test_test """
pred = model.predict(test_set)
pred_class = np.argmax(pred, axis=1)
pred_class
In the comments of this Question you can see that shuffle applies to the base dataset, and such is propagated to the references in the train, test and validation sets.
I would recommend to create 3 distinct datasets, using (e.g.) sklearn.model_selection.train_test_split on the original data before tf.data.Dataset.from_tensor_slices on those split tensor slices, so you can use the shuffle on the training dataset only.
I am training CNN with tf.keras. After of saving checkpoint Keras didn't start next epoch
Note:
1)As a saver was used tf.keras.callbacks.ModelCeckpoint
2)For training used fit_generator()
def iterate_minibatches(inputs, targets, batchsize):
assert len(inputs) == len(targets)
indices = np.arange(len(inputs))
np.random.shuffle(indices)
for start_idx in np.arange(0, len(inputs) - batchsize + 1, batchsize):
excerpt = indices[start_idx:start_idx + batchsize]
yield load_images(inputs[excerpt], targets[excerpt])
#Model path
model_path = "C:/Users/Paperspace/Desktop/checkpoints/cp.ckpt"
#saver = tf.train.Saver(max_to_keep=3)
cp_callback = tf.keras.callbacks.ModelCheckpoint(model_path,
verbose=1,
save_weights_only=True,
period=2)
tb_callback =TensorBoard(log_dir="./Graph/{}".format(time()))
batch_size = 750
history = model.fit_generator(generator=iterate_minibatches(X_train, Y_train,batch_size),
validation_data=iterate_minibatches(X_test, Y_test, batch_size),
# validation_data=None,
steps_per_epoch=len(X_train)//batch_size,
validation_steps=len(X_test)//batch_size,
verbose=1,
epochs=30,
callbacks=[cp_callback,tb_callback]
)
Actual result it stops training without any issue.
Expected result to go next epoch.
**Log**
Epoch 1/30
53/53 [==============================] - 919s 17s/step - loss: 1.2445 - acc: 0.0718
426/426 [==============================] - 7058s 17s/step - loss: 1.7877 - acc: 0.0687 - val_loss: 1.2445 - val_acc: 0.0718
Epoch 2/30
WARNING:tensorflow:Your dataset iterator ran out of data.
Epoch 00002: saving model to C:/Users/Paperspace/Desktop/checkpoints/cp.ckpt
WARNING:tensorflow:This model was compiled with a Keras optimizer (<tensorflow.python.keras.optimizers.Adam object at 0x0000023A913DE470>) but is being saved in TensorFlow format with `save_weights`. The model's weights will be saved, but unlike with TensorFlow optimizers in the TensorFlow format the optimizer's state will not be saved.
Consider using a TensorFlow optimizer from `tf.train`.
WARNING:tensorflow:From C:\Users\Paperspace\Anaconda3\lib\site-packages\tensorflow\python\keras\engine\network.py:1436: update_checkpoint_state (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.train.CheckpointManager to manage checkpoints rather than manually editing the Checkpoint proto.
0/426 [..............................] - ETA: 0s - loss: 0.0000e+00 - acc: 0.0687 - val_loss: 0.0000e+00 - val_acc: 0.0000e+00
On first look, your generator looks incorrect. Keras generators need a while True: loop in them. Maybe this will work for you
def iterate_minibatches(inputs, targets, batchsize):
assert len(inputs) == len(targets)
indices = np.arange(len(inputs))
np.random.shuffle(indices)
while True:
start = 0
end = batchsize
while start < len(inputs):
excerpt = indices[start:end]
yield load_images(inputs[excerpt], targets[excerpt])
start += batchsize
end += batchsize
A Keras generator has to yield batches in an infinite loop. This change should work, otherwise you can follow a tutorial like this.
def iterate_minibatches(inputs, targets, batchsize):
assert len(inputs) == len(targets)
while True:
indices = np.arange(len(inputs))
np.random.shuffle(indices)
for start_idx in np.arange(0, len(inputs) - batchsize + 1, batchsize):
excerpt = indices[start_idx:start_idx + batchsize]
yield load_images(inputs[excerpt], targets[excerpt])
My Keras model below with callback is giving the following output during training.
from keras.callbacks import ModelCheckpoint
checkpoint = ModelCheckpoint('main_model_weights_new.h5', monitor='val_loss', verbose=1,
save_best_only=False, mode='auto',save_weights_only=True)
import pandas
pandas.DataFrame(model.fit(trainX, trainY, epochs=200, batch_size=100,
validation_data=(testX, testY),
callbacks= [checkpoint]).history).to_csv("history.csv")
I was expecting to see the train loss, train accuracy, valid_loss and valid_accuracy. But as shown below it appears there is one more train_loss and train accuracy, output along with the Valid Loss and Valid Accuracy. Can anyone explain me which one to consider as train_loss here?
Output:
Epoch 1/200 2800/2810 [============================>.] - ETA: 0s - loss: 29.7255 - dense_2_loss_1: 3.9492 - dense_2_loss_2: 5.5785 -
dense_2_loss_3: 5.5198 - dense_2_loss_4: 5.6908 - dense_2_loss_5:
4.9863 - dense_2_loss_6: 4.0008 - dense_2_acc_1: 0.1711 - dense_2_acc_2: 0.0836 - dense_2_acc_3: 0.0821 - dense_2_acc_4: 0.1200
- dense_2_acc_5: 0.2393 - dense_2_acc_6: 0.4171Epoch 00000: saving model to main_model_weights_new.h5 2810/2810
[==============================] - 62s - loss: 29.7213 -
dense_2_loss_1: 3.9471 - dense_2_loss_2: 5.5732 - dense_2_loss_3:
5.5226 - dense_2_loss_4: 5.6907 - dense_2_loss_5: 4.9885 - dense_2_loss_6: 3.9992 - dense_2_acc_1: 0.1715 - dense_2_acc_2: 0.0843
- dense_2_acc_3: 0.0822 - dense_2_acc_4: 0.1199 - dense_2_acc_5: 0.2388 - dense_2_acc_6: 0.4167 - val_loss: 31.5189 - val_dense_2_loss_1: 3.6305 - val_dense_2_loss_2: 6.3004 -
val_dense_2_loss_3: 5.9689 - val_dense_2_loss_4: 5.5387 -
val_dense_2_loss_5: 4.9914 - val_dense_2_loss_6: 5.0890 -
val_dense_2_acc_1: 0.2982 - val_dense_2_acc_2: 0.0351 -
val_dense_2_acc_3: 0.0351 - val_dense_2_acc_4: 0.1228 -
val_dense_2_acc_5: 0.2456 - val_dense_2_acc_6: 0.4035 Epoch 2/200