I have two netcdf files (temperature, Humidity) and I would like to use them in a neural network to predict whether it will snow or not. The netcdfs have the format time* lat * lon*value I have a dataframe with the binary result (snow/no snow) for all times.
I have read the blog:
https://www.noahbrenowitz.com/post/loading_netcdfs/
on how to import netcdf in tensorflow but I am confused on how to use them as predictors in tensorflow with my response variable which will be a simple array.
Any idea?
if everything was in a dataframe it would look sth like (as features I created artificial data):
#Split data to test/train
labels=df['Snow']
features = df.iloc[:,2:10]
X=features
y=np.ravel(labels)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
scaler = StandardScaler().fit(X_train)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)
model = Sequential()
model.add(Dense(6, activation='relu', input_shape=(8,)))
model.add(Dense(6, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy',
optimizer='sgd',
metrics=['accuracy'])
model.fit(X_train, y_train,epochs=8, batch_size=1, verbose=1)
y_pred = model.predict(X_test)
score = model.evaluate(X_test, y_test,verbose=1)
Related
I created a simple neural network for understanding how group convolutions can reduce the number of parameters. But when I use the groups parameter in the second convolution layer, I am getting an unimplemented error. However, when groups parameter is not used, everything works fine. Why when using groups parameter, it throws unimplemented error? Does that mean group convolution is not available in keras api?
import tensorflow
from tensorflow.keras.datasets import mnist
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Conv2D,Reshape,MaxPooling2D
from tensorflow.keras.utils import to_categorical
import numpy as np
num_classes = 10
a = np.random.randint(low=0,high=255,size=(100,28,28,1))
b = np.random.randint(low=0,high=10,size=(100,7,7))
a = a.astype('float32')
a = a/255
X_train, Y_train = a[:80], b[:80]
X_test, Y_test = a[80:], b[80:]
num_classes=10
Y_train = to_categorical(Y_train, num_classes)
Y_test = to_categorical(Y_test, num_classes)
# Create the model
model = Sequential()
model.add(Conv2D(8, kernel_size=(3,3),input_shape=(28,28,1),padding='same'))
model.add(Conv2D(8, kernel_size=(3,3),groups=4,input_shape=(28,28,1),padding='same'))
# model.add(Dense(10, input_shape=input_shape, activation='relu'))
model.add(Dense(50, activation='relu'))
model.add(Dense(num_classes, activation='softmax'))
model.add(MaxPooling2D())
model.add(MaxPooling2D())
# model.add(Reshape(target_shape=(10,)))
model.summary()
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(X_train, Y_train, epochs=10, batch_size=250, verbose=1, validation_split=0.2)
# model.save_weights("model.h5")
# # Test the model after training
# test_results = model.evaluate(X_test, Y_test, verbose=1)
# print(f'Test results - Loss: {test_results[0]} - Accuracy: {test_results[1]}%')
Error
UnimplementedError: Fused conv implementation does not support grouped convolutions for now.
[[node sequential_38/conv2d_37/BiasAdd (defined at <ipython-input-42-e7c1c931a421>:50) ]] [Op:__inference_train_function_8596]
Function call stack:
train_function
Here is the colab file for your code. According to the doc
A positive integer specifying the number of groups in which the input is split along the channel axis. Each group is convolved separately with filters / groups filters. The output is the concatenation of all the groups results along the channel axis. Input channels and filters must both be divisible by groups.
In your code, I found no conflict with that. It should work. Otherwise, it may issue with something else.
I have a simple MLP built in Keras. The shapes of my inputs are:
X_train.shape - (6, 5)
Y_train.shape - 6
Create the model
model = Sequential()
model.add(Dense(32, input_shape=(X_train.shape[0],), activation='relu'))
model.add(Dense(Y_train.shape[0], activation='softmax'))
# Compile and fit
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(X_train, Y_train, epochs=10, batch_size=1, verbose=1, validation_split=0.2)
# Get output vector from softmax
output = model.layers[-1].output
This gives me the error:
ValueError: Error when checking input: expected dense_1_input to have shape (6,) but got array with shape (5,).
I have two questions:
Why do I get the above error and how can I solve it?
Is output = model.layers[-1].output the way to return the softmax vector for a given input vector? I haven't ever done this in Keras.
in the input layer use input_shape=(X_train.shape[1],) while your last layer has to be a dimension equal to the number of classes to predict
the way to return the softmax vector is model.predict(X)
here a complete example
n_sample = 5
n_class = 2
X = np.random.uniform(0,1, (n_sample,6))
y = np.random.randint(0,n_class, n_sample)
model = Sequential()
model.add(Dense(32, input_shape=(X.shape[1],), activation='relu'))
model.add(Dense(n_class, activation='softmax'))
# Compile and fit
model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(X, y, epochs=10, batch_size=1, verbose=1)
# Get output vector from softmax
model.predict(X)
I looked at various responses already but I dont understand why I am constantly getting (10, 5).
Why is it asking for a shape of (10,5)? Where is it even getting that number from?
I am under the impression that the shape of the input data should be ("sample_size", "steps or time_len", "channels or feat_size") => (3809, 49, 5).
I am also under the impression that the input shape for Conv1D layer should be ("steps or time_len", "channels or feat_size").
Am I misunderstanding something?
My input data looks something like this:
There is a total of 49 days, 5 data points per each day. There is a total of 5079 sample size. 75% of the data for training, 25% for validation. 10 possible prediction output answers.
x_train, x_test, y_train, y_test = train_test_split(np_train_data, np_train_target, random_state=0)
print(x_train.shape)
x_train = x_train.reshape(x_train.shape[0], round(x_train.shape[1]/5), 5)
x_test = x_test.reshape(x_test.shape[0], round(x_test.shape[1]/5), 5)
print(x_train.shape)
input_shape = (round(x_test.shape[1]/5), 5)
model = Sequential()
model.add(Conv1D(100, 2, activation='relu', input_shape=input_shape))
model.add(MaxPooling1D(3))
model.add(Conv1D(100, 2, activation='relu'))
model.add(GlobalAveragePooling1D())
model.add(Dropout(0.5))
model.add(Dense(49, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(x_train, y_train, batch_size=64, epochs=2, validation_data=(x_test, y_test))
print(model.summary())
I get this error:
Print out of layers
You are dividing by 5 twice. Here you are reshaping your data, which is necessary contrary to what the other answer says:
x_train = x_train.reshape(x_train.shape[0], round(x_train.shape[1]/5), 5)
x_test = x_test.reshape(x_test.shape[0], round(x_test.shape[1]/5), 5)
This already takes care of "dividing the time by 5". But here you are defining the input shape to the model, dividing by 5 again:
input_shape = (round(x_test.shape[1]/5), 5)
Simply use
input_shape = (x_test.shape[1], 5)
instead! Note that because this shape is called after the reshape, it already refers to the correct one, with the time dimension divided by 5.
You are using Conv1D, but trying, by reshaping, represent your data in 2D - that make a problem. Try to skip the part with reshaping, so your input will be a 1 row with 49 values:
x_train, x_test, y_train, y_test = train_test_split(np_train_data, np_train_target, random_state=0)
print(x_train.shape)
input_shape = (x_test.shape[1], 1)
model = Sequential()
model.add(Conv1D(100, 2, activation='relu', input_shape=input_shape))
model.add(MaxPooling1D(3))
model.add(Conv1D(100, 2, activation='relu'))
model.add(GlobalAveragePooling1D())
model.add(Dropout(0.5))
model.add(Dense(49, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(x_train, y_train, batch_size=64, epochs=2, validation_data=(x_test, y_test))
Y_train = to_categorical(Y_train, num_classes = 10)#
random_seed = 2
X_train,X_val,Y_train,Y_val = train_test_split(X_train, Y_train, test_size = 0.1, random_state=random_seed)
Y_train.shape
model = Sequential()
model.add(Flatten())
model.add(Dense(64, activation='relu'))
model.add(Dense(128, activation='relu'))
model.add(Dense(10, activation='softmax'))
model.compile(optimizer = 'adam', loss = 'sparse_categorical_crossentropy',metrics=['accuracy'])
model.fit(X_train, Y_train, batch_size = 86, epochs = 3,validation_data = (X_val, Y_val), verbose =2)
I have to classify the MNIST data into 10 classes. I am converting the Y_train into one hot encoded array. I have gone through a number of answers but none have helped. Kindly guide me in this regard as I am a novice in ML and neural network.
It seems there is no need to use model.add(Flatten()) in your first layer. Instead of doing so, you can use a dense layer with a specific input size like: model.add(Dense(64, input_shape=your_input_shape, activation="relu").
To ensure this issue happens because of the layers, you can check whether to_categorical() function works alone with jupyter notebook.
Updated Answer
Before the model, you should reshape your model. In that case 28*28 to 784.
train_images = train_images.reshape((-1, 784))
test_images = test_images.reshape((-1, 784))
I also suggest to normalize the data that could be done by simply dividing the images to 255
After that step you should create your model.
model = Sequential([
Dense(64, activation='relu', input_shape=(784,)),
Dense(64, activation='relu'),
Dense(10, activation='softmax'),
])
Have you noticed input_shape=(784,) That is the shape of your flattened input.
Last step, compiling and fitting.
model.compile(
optimizer='adam',
loss='categorical_crossentropy',
metrics=['accuracy'],
)
model.fit(
train_images,
train_labels,
epochs=10,
batch_size=16,
)
What you do is you have just flattened the input layer without feeding the network with an input. That's why you experience an issue. The point is you should manually reshape your inputs and feed forward to the Dense() layers with parameter input_shape
I'm implementing this simple neural network, with these inputs data:
x_train = np.asarray(x_train)
y_train = np.asarray(y_train)
x_test = np.asarray(x_test)
y_test = np.asarray(y_test)
After have defined the network's structure:
model = Sequential()
model.add(Dense(20, input_dim=5, init='normal', activation='sigmoid'))
model.add(Dense(1, init='normal', activation='sigmoid'))
I run this to train and evaluate the NN:
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
history = model.fit(x_train, y_train, nb_epoch=10, validation_split=0.2)
and I always get the same result from model.fit( ... ) :
32/143 [=====>........................] - ETA: 0s.
It seems that doesn't work at all, despite I obtain consistent results on training and validation. How does I have to interpreter this stationary result about model. fit output?