Obtaining multiple plots of a given data - pandas

I have this data:
s_result = [{'time': array([ 0. , 0.1, 0.2, ..., 299.7, 299.8, 299.9]), 'I': array([10., 10., 10., ..., 0., 0., 0.]), 'S': array([60., 60., 60., ..., 0., 0., 0.]), 'M': array([40., 40., 40., ..., 0., 0., 0.]), 'R': array([0., 0., 0., ..., 1., 1., 1.]), 'P1': array([ 0., 0., 0., ..., 19., 19., 19.]), 'D1': array([ 0., 0., 0., ..., 81., 81., 81.]), 'P2': array([0., 0., 0., ..., 0., 0., 0.]), 'D2': array([0., 0., 0., ..., 0., 0., 0.]), 'P3': array([0., 0., 0., ..., 0., 0., 0.]), 'D3': array([0., 0., 0., ..., 0., 0., 0.]), 'P4': array([0., 0., 0., ..., 0., 0., 0.]), 'D4': array([0., 0., 0., ..., 0., 0., 0.]), 'P5': array([0., 0., 0., ..., 0., 0., 0.]), 'D5': array([0., 0., 0., ..., 0., 0., 0.]), 'P6': array([0., 0., 0., ..., 0., 0., 0.]), 'D6': array([0., 0., 0., ..., 0., 0., 0.]), 'P7': array([0., 0., 0., ..., 0., 0., 0.]), 'D7': array([0., 0., 0., ..., 0., 0., 0.]), 'P8': array([0., 0., 0., ..., 0., 0., 0.]), 'D8': array([0., 0., 0., ..., 0., 0., 0.]), 'P9': array([0., 0., 0., ..., 0., 0., 0.]), 'D9': array([0., 0., 0., ..., 0., 0., 0.])}, {'time': array([ 0. , 0.1, 0.2, ..., 299.7, 299.8, 299.9]), 'I': array([10., 10., 10., ..., 0., 0., 0.]), 'S': array([60., 60., 60., ..., 0., 0., 0.]), 'M': array([40., 40., 40., ..., 0., 0., 0.]), 'R': array([0., 0., 0., ..., 0., 0., 0.]), 'P1': array([ 0., 0., 0., ..., 20., 20., 20.]), 'D1': array([ 0., 0., 0., ..., 80., 80., 80.]), 'P2': array([0., 0., 0., ..., 0., 0., 0.]), 'D2': array([0., 0., 0., ..., 0., 0., 0.]), 'P3': array([0., 0., 0., ..., 0., 0., 0.]), 'D3': array([0., 0., 0., ..., 0., 0., 0.]), 'P4': array([0., 0., 0., ..., 0., 0., 0.]), 'D4': array([0., 0., 0., ..., 0., 0., 0.]), 'P5': array([0., 0., 0., ..., 0., 0., 0.]), 'D5': array([0., 0., 0., ..., 0., 0., 0.]), 'P6': array([0., 0., 0., ..., 0., 0., 0.]), 'D6': array([0., 0., 0., ..., 0., 0., 0.]), 'P7': array([0., 0., 0., ..., 0., 0., 0.]), 'D7': array([0., 0., 0., ..., 0., 0., 0.]), 'P8': array([0., 0., 0., ..., 0., 0., 0.]), 'D8': array([0., 0., 0., ..., 0., 0., 0.]), 'P9': array([0., 0., 0., ..., 0., 0., 0.]), 'D9': array([0., 0., 0., ..., 0., 0., 0.])}, {'time': array([ 0. , 0.1, 0.2, ..., 299.7, 299.8, 299.9]), 'I': array([10., 10., 10., ..., 0., 0., 0.]), 'S': array([60., 60., 60., ..., 0., 0., 0.]), 'M': array([40., 40., 40., ..., 0., 0., 0.]), 'R': array([0., 0., 0., ..., 0., 0., 0.]), 'P1': array([ 0., 0., 0., ..., 20., 20., 20.]), 'D1': array([ 0., 0., 0., ..., 80., 80., 80.]), 'P2': array([0., 0., 0., ..., 0., 0., 0.]), 'D2': array([0., 0., 0., ..., 0., 0., 0.]), 'P3': array([0., 0., 0., ..., 0., 0., 0.]), 'D3': array([0., 0., 0., ..., 0., 0., 0.]), 'P4': array([0., 0., 0., ..., 0., 0., 0.]), 'D4': array([0., 0., 0., ..., 0., 0., 0.]), 'P5': array([0., 0., 0., ..., 0., 0., 0.]), 'D5': array([0., 0., 0., ..., 0., 0., 0.]), 'P6': array([0., 0., 0., ..., 0., 0., 0.]), 'D6': array([0., 0., 0., ..., 0., 0., 0.]), 'P7': array([0., 0., 0., ..., 0., 0., 0.]), 'D7': array([0., 0., 0., ..., 0., 0., 0.]), 'P8': array([0., 0., 0., ..., 0., 0., 0.]), 'D8': array([0., 0., 0., ..., 0., 0., 0.]), 'P9': array([0., 0., 0., ..., 0., 0., 0.]), 'D9': array([0., 0., 0., ..., 0., 0., 0.])}]
I intend to work on only M as follows:
for index in range(0, 3):
x_stochastic = s_result[index]['M']
x_stochastic=((s_result['M'][0]-s_result['M'][:])/s_result['M'][0])
plt.plot(s_trajectory['time'],x_stochastic , 'r')
plt.xlabel('Time')
plt.ylabel('Monomer Conversion,X')
The expected outcome is 3 different trajectories as shown in the above data. But this is what got graphically:

I couldn't get your code as provided to run. Furthermore in the data snipped you posted all three M Value series are indistinguishable.
I've taken the liberty to modify the data to
s_result = [{'time': np.array([0,100,200,300]),
'M': np.array([40,40,30,0])},
{'time': np.array([0,100,200,300]),
'M': np.array([40,10,5,0])},
]
The corrected loop
for index in range(0, len(s_result)):
x_stochastic = s_result[index]
x_stochastic=((x_stochastic['M'][0]-x_stochastic['M'][:])/x_stochastic['M'][0])
plt.plot(s_result[index]['time'],x_stochastic , 'r')
plt.xlabel('Time')
plt.ylabel('Monomer Conversion,X')
then produces two distinct traces as desired.

Related

merge two thresholds for two 3D arrays into a list

I have the first 3D array of size (50,250,250) that includes data points (1,2,3,4,5). I set up a threshold that is 3, where the data points above should equal to 1 and below it equal to 0. the only exception is when the data points are equal to 3, it has to test the second threshold (threshold1=50) that is based on the second 3D array of size (50,250,250). my equation is how to include the two thresholds in my code! In other words, the for loop will check every datapoint in array 1 and perform the first threshold testing, if the datapoint is equal to 3, the for loop should check the counterpart of that datapoint in the second array for the second threshold testing! I have tried the below code, but the results did not make sense
res1=[]
f1=numpy.ones((250, 250))
threshold=3
threshold1=30
for i in array1:
i = i.data
ii= f1*i
ii[ii < threshold] = 0
ii[ii > threshold] = 1
res1.append(ii)
if ii[ii == threshold]:
for j in array2:
j = j.data
jj[jj < threshold1] = 0
jj[jj > threshold1] = 1
res1.append(jj)
Array1:
array([[[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
...,
[3., 3., 3., ..., 0., 0., 0.],
[3., 3., 3., ..., 0., 0., 0.],
[3., 3., 3., ..., 0., 0., 0.]],
[[0., 0., 0., ..., 0., 0., 1.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
...,
[3., 3., 3., ..., 0., 0., 0.],
[3., 3., 3., ..., 0., 0., 0.],
[3., 3., 3., ..., 0., 0., 0.]],
Array2:[[ nan, nan, nan, ..., nan,
0.9839769, 1.7042577],
[ nan, nan, nan, ..., nan,
nan, nan],
[ nan, nan, nan, ..., 3.2351596,
2.0924768, 1.7604152],
...,
[ nan, nan, nan, ..., 158.48865 ,
158.48865 , 125.888 ],
[ nan, nan, nan, ..., 158.48865 ,
158.48865 , 158.48865 ],
[ nan, nan, nan, ..., 125.88556 ,
158.48865 , 158.48865 ]],
the produced list (rest1)
`[array([[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
...,
[1., 1., 1., ..., 0., 0., 0.],
[1., 1., 1., ..., 0., 0., 0.],
[1., 1., 1., ..., 0., 0., 0.]]),
array([[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
...,
[1., 1., 1., ..., 0., 0., 0.],
[1., 1., 1., ..., 0., 0., 0.],
[1., 1., 1., ..., 0., 0., 0.]]),
array([[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],`
IIUC, for your second if condition, you are trying to see whether there is at least a 3 value in that array1, and then you will choose that 2D array of the same position. In that case, you should use in operator.
for i in range(len(array1)):
if threshold in array1[i]:
array2[i][array2[i] < threshold1] = 0
array2[i][array2[i] > threshold1] = 1
res1.append(array2[i])
else:
array1[i][array1[i] < threshold] = 0
array1[i][array1[i] > threshold] = 1
res1.append(array1[i])
The above method is a bit lengthy for numpy. There's a numpy way to do this, too.
array1[array1 < threshold] = 0
array1[array1 > threshold] = 1
array2_condition = np.unique(np.argwhere(array1 == 3)[:,0]) # return the index of array1 if 3 in array1
chosen_array2 = array2[array2_condition]
chosen_array2[chosen_array2 < threshold1] = 0
chosen_array2[chosen_array2 > threshold1] = 1
array2[array2_condition] = chosen_array2 # if you still want array2 values to be changed
res1 = array1
res1[array2_condition] = chosen_array2 # Final result
Update
As was mentioned by the OP, every 2D array has at least a 3 in it. So, the array2_condition is not applicable. Instead, we will modify the array2_condition and use a for loop to change the elements.
res1 = array1
res1[res1 < threshold] = 0
res1[res1 > threshold] = 1
array2_condition = np.argwhere(array1 == 3)
for data in array2_condition:
if array2[tuple(data)] > threshold1:
res1[tuple(data)] = 1
elif array2[tuple(data)] < threshold1:
res12[tuple(data)] = 0

One-hot encode labels in keras

I have a set of integers from a label column in a CSV file - [1,2,4,3,5,2,..]. The number of classes is 5 ie range of 1 to 6. I want to one-hot encode them using the below code.
y = df.iloc[:,10].values
y = tf.keras.utils.to_categorical(y, num_classes = 5)
y
But this code gives me an error
IndexError: index 5 is out of bounds for axis 1 with size 5
How can I fix this?
If you use tf.keras.utils.to_categorical to one-hot the label vector, the integers should start from 0 to num_classes, source. In your case, you should do as follows
import tensorflow as tf
import numpy as np
a = np.array([1,2,4,3,5,2,4,2,1])
y_tf = tf.keras.utils.to_categorical(a-1, num_classes = 5)
y_tf
array([[1., 0., 0., 0., 0.],
[0., 1., 0., 0., 0.],
[0., 0., 0., 1., 0.],
[0., 0., 1., 0., 0.],
[0., 0., 0., 0., 1.],
[0., 1., 0., 0., 0.],
[0., 0., 0., 1., 0.],
[0., 1., 0., 0., 0.],
[1., 0., 0., 0., 0.]], dtype=float32)
or, you can use pd.get_dummies,
import pandas as pd
import numpy as np
a = np.array([1,2,4,3,5,2,4,2,1])
a_pd = pd.get_dummies(a).astype('float32').values
a_pd
array([[1., 0., 0., 0., 0.],
[0., 1., 0., 0., 0.],
[0., 0., 0., 1., 0.],
[0., 0., 1., 0., 0.],
[0., 0., 0., 0., 1.],
[0., 1., 0., 0., 0.],
[0., 0., 0., 1., 0.],
[0., 1., 0., 0., 0.],
[1., 0., 0., 0., 0.]], dtype=float32)

Keras: result of model.evaluate() stays high with all the weights and biases being 0

I created a VGG16 model using Keras application (TensorFlow backend). Then I wanted to change part of those weights and then test the accuracy of this modified model. To be direct and intuitive, I changed ALL the weights and biases in ALL layers to 0 like this:
model = VGG16(weights='imagenet', include_top=True)
# here is the test data and label containing 10 pictures I created.
data = np.load('./10_random_samples_array.npz')
data, label = data["X"], data["Y"]
# Modify the weights to zero
for z in [1, 2, 4, 5, 7, 8, 9, 11, 12, 13, 15, 16, 17]: # Conv layers
weight_bias = model.layers[z].get_weights()
shape_weight = np.shape(weight_bias[0])
shape_bias = np.shape(weight_bias[1])
weight_bias[0] = np.zeros(shape=(shape_weight[0],shape_weight[1],shape_weight[2],shape_weight[3]))
weight_bias[1] = np.zeros(shape=(shape_bias[0],))
model.layers[z].set_weights(weight_bias)
for z in [20,21,22]: # FC layers
weight_bias = model.layers[z].get_weights()
shape_weight = np.shape(weight_bias[0])
print(z, shape_weight)
shape_bias = np.shape(weight_bias[1])
weight_bias[0] = np.zeros(shape=(shape_weight[0],shape_weight[1],))
weight_bias[1] = np.zeros(shape=(shape_bias[0],))
model.layers[z].set_weights(weight_bias)
model.compile(loss='binary_crossentropy',
optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),
metrics=['accuracy'])
# To check if the weights have been modified.
print(model.layers[1].get_weights())
loss, acc = model.evaluate(data, label, verbose=1)
print(acc)
Then I got result like this:
[array([[[[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.]],
...(All zero, I omit them)
[[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.],
[0., 0., 0., ..., 0., 0., 0.]]]], dtype=float32),
array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)]
10/10 [==============================] - 2s 196ms/step
0.9989999532699585
Firstly, You can tell that all the weights and biases have already been changed to 0 but the accuracy still stays very high. That is unreasonable.(The original result returned by model.evaluate() is 0.9993000030517578)
Secondly, I used only 10 pictures as my test dataset. The result must be a decimal with only one digit after the point. But I got 0.9989999532699585.
I also tried to modify all weights only in Conv1-1 to zero and the result is also 0.9989999532699585. It seems that it is the minimum result. Is there something wrong with my model? Or the weights cannot be modified in this way? Or model.evaluate() doesn't work as I suppose?

Error computing KL divergence in Scipy

I am trying to calculate KL divergence using the entropy function of scipy.
My p is:
array([[ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1.],
[ 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[ 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[ 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[ 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])
and q is:
array([[ 0.05242718, 0.04436347, 0.04130855, 0.04878344, 0.04310538,
0.02856853, 0.03303122, 0.02517992, 0.08525434, 0.03450324,
0.14580068, 0.1286993 , 0.28897473],
[ 0.65421444, 0.11592199, 0.0642645 , 0.02989768, 0.01385762,
0.01756484, 0.01024294, 0.00891479, 0.01140301, 0.00718939,
0.00938009, 0.01070139, 0.04644726],
[ 0.65984136, 0.13251236, 0.06345234, 0.02891162, 0.02429709,
0.02025307, 0.01073064, 0.01170066, 0.00678652, 0.00703361,
0.00560414, 0.00651137, 0.02236522],
[ 0.32315928, 0.23900077, 0.05460232, 0.03953635, 0.02901102,
0.01294443, 0.02372061, 0.02092882, 0.01188251, 0.01377188,
0.02976672, 0.05854314, 0.14313218],
[ 0.7717858 , 0.09692616, 0.03415596, 0.01713088, 0.01108141,
0.0128005 , 0.00847301, 0.01049734, 0.0052889 , 0.00514799,
0.00442508, 0.00485477, 0.01743218]], dtype=float32)
When I do:
entropy(p[0],q[0])
I am getting the following error:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-201-563ea7d4decf> in <module>()
4 print('p0:',p[0])
5 print('q0:',q[0])
----> 6 entropy(p[0],q[0])
/Users/freelancer/anaconda/envs/py35/lib/python3.5/site-packages/matplotlib/mlab.py in entropy(y, bins)
1570 y = np.zeros((len(x)+2,), x.dtype)
1571 y[1:-1] = x
-> 1572 dif = np.diff(y)
1573 up = (dif == 1).nonzero()[0]
1574 dn = (dif == -1).nonzero()[0]
/Users/freelancer/anaconda/envs/py35/lib/python3.5/site-packages/numpy/lib/function_base.py in histogram(a, bins, range, normed, weights, density)
781 if (np.diff(bins) < 0).any():
782 raise ValueError(
--> 783 'bins must increase monotonically.')
784
785 # Initialize empty histogram
ValueError: bins must increase monotonically.
Why is it?
This works with the example arrays:
import scipy as sp
sp.stats.entropy(p[0], q[0])
Looking at the stack trace in the error massage, it becomes apparent that you did not call scipy's entropy function but matplotlib's entropy, which works differently.
Here is the relevant part:
/Users/freelancer/anaconda/envs/py35/lib/python3.5/site-packages/matplotlib/mlab.pyin entropy(y, bins)

Logical addressing numpy mess up with other matrices

I have just found a problem and I don't know if it is meant to be this way or I am just doing it wrong. When I use logical addressing in a numpy matrix to change all the values of a matrix that are, say, equal to a 1. All other matrices that somehow have something to do with this matrix will also be modified.
In [1]: import numpy as np
In [2]: from numpy import matrix as mtx
In [3]: A=mtx(np.eye(6))
In [4]: A
Out[4]:
matrix([[ 1., 0., 0., 0., 0., 0.],
[ 0., 1., 0., 0., 0., 0.],
[ 0., 0., 1., 0., 0., 0.],
[ 0., 0., 0., 1., 0., 0.],
[ 0., 0., 0., 0., 1., 0.],
[ 0., 0., 0., 0., 0., 1.]])
In [5]: B=A
In [6]: C=B
In [7]: D=C
In [8]: A[A==1]=5
In [9]: A
Out[9]:
matrix([[ 5., 0., 0., 0., 0., 0.],
[ 0., 5., 0., 0., 0., 0.],
[ 0., 0., 5., 0., 0., 0.],
[ 0., 0., 0., 5., 0., 0.],
[ 0., 0., 0., 0., 5., 0.],
[ 0., 0., 0., 0., 0., 5.]])
In [10]: B
Out[10]:
matrix([[ 5., 0., 0., 0., 0., 0.],
[ 0., 5., 0., 0., 0., 0.],
[ 0., 0., 5., 0., 0., 0.],
[ 0., 0., 0., 5., 0., 0.],
[ 0., 0., 0., 0., 5., 0.],
[ 0., 0., 0., 0., 0., 5.]])
In [11]: C
Out[11]:
matrix([[ 5., 0., 0., 0., 0., 0.],
[ 0., 5., 0., 0., 0., 0.],
[ 0., 0., 5., 0., 0., 0.],
[ 0., 0., 0., 5., 0., 0.],
[ 0., 0., 0., 0., 5., 0.],
[ 0., 0., 0., 0., 0., 5.]])
In [12]: D
Out[12]:
matrix([[ 5., 0., 0., 0., 0., 0.],
[ 0., 5., 0., 0., 0., 0.],
[ 0., 0., 5., 0., 0., 0.],
[ 0., 0., 0., 5., 0., 0.],
[ 0., 0., 0., 0., 5., 0.],
[ 0., 0., 0., 0., 0., 5.]])
Can anyone tell me what am I doing wrong? is this a bug?
This is not a bug. Saying B=A in python means that both B and A point to the same object. You need to copy the matrix.
>>> import numpy as np
>>> from numpy import matrix as mtx
>>> A = mtx(np.eye(6))
>>> B = A.copy()
>>> C = A
#Check memory locations.
>>> id(A)
19608352
>>> id(C)
19608352 #Same object as A
>>> id(B)
19607992 #Different object then A
>>> A[A==1] = 5
>>> B #B is a different object then A
matrix([[ 1., 0., 0., 0., 0., 0.],
[ 0., 1., 0., 0., 0., 0.],
[ 0., 0., 1., 0., 0., 0.],
[ 0., 0., 0., 1., 0., 0.],
[ 0., 0., 0., 0., 1., 0.],
[ 0., 0., 0., 0., 0., 1.]])
>>> C #C is the same object as A
matrix([[ 5., 0., 0., 0., 0., 0.],
[ 0., 5., 0., 0., 0., 0.],
[ 0., 0., 5., 0., 0., 0.],
[ 0., 0., 0., 5., 0., 0.],
[ 0., 0., 0., 0., 5., 0.],
[ 0., 0., 0., 0., 0., 5.]])
The same issue can be seen with python list:
>>> A = [5,3]
>>> B = A
>>> B[0] = 10
>>> A
[10, 3]
Note that this is different then returning a numpy view as in this case:
>>> A = mtx(np.eye(6))
>>> B = A[0] #B is a view and now points to the first row of A
>>> id(A)
28088720
>>> id(B) #Different objects!
28087568
#B still points to the memory location of A's first row, but through numpy trickery
>>> B
matrix([[ 1., 0., 0., 0., 0., 0.]])
>>> B *= 5 #In place multiplication, updates B which is the same as A's first row
>>> A
matrix([[ 5., 0., 0., 0., 0., 0.],
[ 0., 1., 0., 0., 0., 0.],
[ 0., 0., 1., 0., 0., 0.],
[ 0., 0., 0., 1., 0., 0.],
[ 0., 0., 0., 0., 1., 0.],
[ 0., 0., 0., 0., 0., 1.]])
As the view B points to the first row of A, A is changed. Now lets force a copy.
>>> B = B*10 #Assigns B*10 to a different chunk of memory
>>> A
matrix([[ 5., 0., 0., 0., 0., 0.],
[ 0., 1., 0., 0., 0., 0.],
[ 0., 0., 1., 0., 0., 0.],
[ 0., 0., 0., 1., 0., 0.],
[ 0., 0., 0., 0., 1., 0.],
[ 0., 0., 0., 0., 0., 1.]])
>>> B
matrix([[ 50., 0., 0., 0., 0., 0.]])