[UPDATED] I'm working on a nonlinear ODEs system optimization and fitting it to experimental data. I have a system of 5 model ODEs which must be optimized by 17 parameters. My approach is to calculate the differences between solved ODEs and experimental data - function Differences, then use leastsq solver to minimize diferences and find the optimal parameters, as below code:
//RHSs of ODEs to be fitted:
function dx=model3(t,x,Kap,Ksa,Ko,Ks,Kia,Kis,p_Amax,q_Amax,qm,q_Smax,Yas,Yoa,Yxa,Yem,Yos,Yxsof,H)
X=x(1);
S=x(2);
A=x(3);
DO=x(4);
V=x(5);`
qs=((q_Smax*S/(S+Ks))*Kia/(Kia+A));
qsof=(p_Amax*qs/(qs+Kap));
qsox=(qs-qsof)*DO/(DO+Ko);
qsa=(q_Amax*A/(A+Ksa))*(Kis/(qs+Kis));
pa=qsof*Yas;
qa=pa-qsa;
qo=(qsox-qm)*Yos+qsa*Yoa;
u=(qsox-qm)*Yem+qsof*Yxsof+qsa*Yxa;
dx(1)=u*X-F*X/V;
dx(2)=(F*(Sf-S)/V)-qs*X;
dx(3)=qsa*X-(F*A/V);
dx(4)=200*(100-DO)-qo*X*H;
dx(5)=F;
endfunction
//experimental data:
//Dat=fscanfMat('dane_exper_III_etap.txt');
Dat = [
0 30 1.4 24.1 99 6884.754
1 35 0.2 23.2 89 6959.754
2 40 0.1 21.6 80 7034.754
3 52 0.1 19.5 67 7109.754
4 61 0.1 18.7 70 7184.754
5 66 0.1 16.4 79 7259.754
6 71 0.1 15 94 7334.754
7 74 0 14.3 100 7409.754
8 76 0 13.8 100 7484.754
9 78 0 13.4 100 7559.754
9.5 79 0 13.2 100 7597.254
10 79 0 13.5 100 7634.754]
t=Dat(:,1);
x_exp(:,1)=Dat(:,2);
x_exp(:,2)=Dat(:,3);
x_exp(:,3)=Dat(:,4);
x_exp(:,4)=Dat(:,5);
x_exp(:,5)=Dat(:,6);
global MYDATA;
MYDATA.t=t;
MYDATA.x_exp=x_exp;
MYDATA.funeval=0;
//calculating differences between calculated values and experimental data:
function f=Differences(k)
global MYDATA
t=MYDATA.t;
x_exp=MYDATA.x_exp;
Kap=k(1); //g/L
Ksa=k(2); //g/L
Ko=k(3); //g/L
Ks=k(4); //g/L
Kia=k(5); //g/L
Kis=k(6); //g/L
p_Amax=k(7); //g/(g*h)
q_Amax=k(8); //g/(g*h)
qm=k(9);
q_Smax=k(10);
Yas=k(11); //g/g
Yoa=k(12);
Yxa=k(13);
Yem=k(14);
Yos=k(15);
Yxsof=k(16);
H=k(17);
x0=x_exp(1,:);
t0=0;
F=75;
Sf=500;
%ODEOPTIONS=[1,0,0,%inf,0,2,10000,12,5,0,-1,-1]
x_calc=ode('rk',x0',t0,t,list(model3,Kap,Ksa,Ko,Ks,Kia,Kis,p_Amax,q_Amax,qm,q_Smax,Yas,Yoa,Yxa,Yem,Yos,Yxsof,H));
diffmat=x_calc'-x_exp;
//column vector of differences (concatenates 4 columns of the difference matrix)
f=diffmat(:);
MYDATA.funeval=MYDATA.funeval+1;
endfunction
// Initial guess
Kap=0.3; //g/L
Ksa=0.05; //g/L
Ko=0.1; //g/L
Ks=0.5; //g/L
Kia=0.5; //g/L
Kis=0.05; //g/L
p_Amax=0.4; //g/(g*h)
q_Amax=0.8; //g/(g*h)
qm=0.2;
q_Smax=0.6;
Yas=0.5; //g/g
Yoa=0.5;
Yxa=0.5;
Yem=0.5;
Yos=1.5;
Yxsof=0.22;
H=1000;
y0=[Kap;Ksa;Ko;Ks;Kia;Kis;p_Amax;q_Amax;qm;q_Smax;Yas;Yoa;Yxa;Yem;Yos;Yxsof;H];
yinf=[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,100];
ysup=[%inf,%inf,%inf,%inf,%inf,%inf,3,3,3,3,3,3,3,3,3,3,10000];
[fopt,xopt,gopt]=leastsq(Differences,'b',yinf,ysup,y0);
Now result is:
0.2994018
0.0508325
0.0999987
0.4994088
0.5081272
0.
0.4004560
0.7050746
0.2774195
0.6068328
0.5
0.4926150
0.4053860
0.5255006
1.5018725
0.2193901
1000.0000
33591.642
Running this script causes such an error:
lsoda-- caution... t (=r1) and h (=r2) are
such that t + h = t at next step
(h = pas). integration continues
where r1 is : 0.5658105345269D+01 and r2 : 0.1884898700920D-17
lsoda-- previous message precedent given i1 times
will no more be repeated
where i1 is : 10
lsoda-- at t (=r1), mxstep (=i1) steps
needed before reaching tout
where i1 is : 500000
where r1 is : 0.5658105345270D+01
Excessive work done on this call (perhaps wrong jacobian type).
at line 27 of function Differences
I understand that problem is on ODEs solving step. Thus, I have tried changing the mxstep, as also solving method type to 'adams','rk', and 'stiff' - none of this solved the problem. Using 'fix' method in ode I get this error:
ode: rksimp exit with state 3.
Please advise how to solve this?
P.S. Experimental data in file 'dane_exper_III_etap.txt':
0 30 1.4 24.1 99 6884.754
1 35 0.2 23.2 89 6959.754
2 40 0.1 21.6 80 7034.754
3 52 0.1 19.5 67 7109.754
4 61 0.1 18.7 70 7184.754
5 66 0.1 16.4 79 7259.754
6 71 0.1 15 94 7334.754
7 74 0 14.3 100 7409.754
8 76 0 13.8 100 7484.754
9 78 0 13.4 100 7559.754
9.5 79 0 13.2 100 7597.254
10 79 0 13.5 100 7634.754
In Scilab leastsq (based on optim) is very poor and doesn't have global convergence properties, unlike ipopt which is available as an atoms module. Install it like this:
--> atomsInstall sci_ipopt
I have modified your script in the following way
keep classical use of ode for this kind of biological kinetics, i.e. "stiff" (which uses BDF method). The Runge-Kutta you were using is very poor as it is an explicit method only for gentle ODEs.
use ipopt instead of leastsq
use a try/catch/end block around the computation of the residual in order to catch failing calls to the ode solver.
use some weight for the residual. You should play with it in order to improve the fit
use a strictly positive lower bound instead of 0, as very low value of some parameters make the ode solver fail.
add a drawing callback that also saves current value of parameters in case where you stop the optimization with ctrl-C.
//RHSs of ODEs to be fitted:
function dx=model3(t,x,Kap,Ksa,Ko,Ks,Kia,Kis,p_Amax,q_Amax,qm,q_Smax,Yas,Yoa,Yxa,Yem,Yos,Yxsof,H)
X=x(1);
S=x(2);
A=x(3);
DO=x(4);
V=x(5);
qs=((q_Smax*S/(S+Ks))*Kia/(Kia+A));
qsof=(p_Amax*qs/(qs+Kap));
qsox=(qs-qsof)*DO/(DO+Ko);
qsa=(q_Amax*A/(A+Ksa))*(Kis/(qs+Kis));
pa=qsof*Yas;
qa=pa-qsa;
qo=(qsox-qm)*Yos+qsa*Yoa;
u=(qsox-qm)*Yem+qsof*Yxsof+qsa*Yxa;
dx(1)=u*X-F*X/V;
dx(2)=(F*(Sf-S)/V)-qs*X;
dx(3)=qsa*X-(F*A/V);
dx(4)=200*(100-DO)-qo*X*H;
dx(5)=F;
endfunction
//calculating differences between calculated values and experimental data:
function [f,x_calc]=Differences(k, t, x_exp)
Kap=k(1); //g/L
Ksa=k(2); //g/L
Ko=k(3); //g/L
Ks=k(4); //g/L
Kia=k(5); //g/L
Kis=k(6); //g/L
p_Amax=k(7); //g/(g*h)
q_Amax=k(8); //g/(g*h)
qm=k(9);
q_Smax=k(10);
Yas=k(11); //g/g
Yoa=k(12);
Yxa=k(13);
Yem=k(14);
Yos=k(15);
Yxsof=k(16);
H=k(17);
x0=x_exp(1,:);
t0=0;
F=75;
Sf=500;
[x_calc]=ode("stiff",x0',t0,t,list(model3,Kap,Ksa,Ko,Ks,Kia,Kis,p_Amax,q_Amax,qm,q_Smax,Yas,Yoa,Yxa,Yem,Yos,Yxsof,H));
diffmat=(x_calc'-x_exp)*residual_weight;
//column vector of differences (concatenates 4 columns of the difference matrix)
f=diffmat(:);
MYDATA.funeval=MYDATA.funeval+1;
endfunction
function [f,g]=normdiff2(k,new_k,t,x_exp)
try
res = Differences(k,t,x_exp)
if argn(1) == 2
JacRes = numderivative(list(Differences,t,x_exp),k)
g = 2*JacRes'*res;
end
f = sum(res.*res)
catch
f=%inf;
g=%inf*ones(k);
end
endfunction
function out=callback(param)
global MYDATA
if isfield(param,"x")
k = param.x;
MYDATA.k = k;
[f,x_calc]=Differences(k,t,x_exp)
plot_weight = diag(1./max(x_exp,'r'));
drawlater
clf
plot(t,x_exp*plot_weight,'-o')
plot(t,x_calc'*plot_weight,'-x')
legend X S A DO X
drawnow
end
out = %t;
endfunction
//experimental data:
//Dat=fscanfMat('dane_exper_III_etap.txt');
Dat = [
0 30 1.4 24.1 99 6884.754
1 35 0.2 23.2 89 6959.754
2 40 0.1 21.6 80 7034.754
3 52 0.1 19.5 67 7109.754
4 61 0.1 18.7 70 7184.754
5 66 0.1 16.4 79 7259.754
6 71 0.1 15 94 7334.754
7 74 0 14.3 100 7409.754
8 76 0 13.8 100 7484.754
9 78 0 13.4 100 7559.754
9.5 79 0 13.2 100 7597.254
10 79 0 13.5 100 7634.754]
t=Dat(:,1);
x_exp(:,1)=Dat(:,2);
x_exp(:,2)=Dat(:,3);
x_exp(:,3)=Dat(:,4);
x_exp(:,4)=Dat(:,5);
x_exp(:,5)=Dat(:,6);
global MYDATA;
MYDATA.funeval=0;
// Initial guess
Kap=0.3; //g/L
Ksa=0.05; //g/L
Ko=0.1; //g/L
Ks=0.5; //g/L
Kia=0.5; //g/L
Kis=0.05; //g/L
p_Amax=0.4; //g/(g*h)
q_Amax=0.8; //g/(g*h)
qm=0.2;
q_Smax=0.6;
Yas=0.5; //g/g
Yoa=0.5;
Yxa=0.5;
Yem=0.5;
Yos=1.5;
Yxsof=0.22;
H=100;
k0 = [Kap;Ksa;Ko;Ks;Kia;Kis;p_Amax;q_Amax;qm;q_Smax;Yas;Yoa;Yxa;Yem;Yos;Yxsof;H];
residual_weight = diag(1./[79,1.4, 24.1, 100, 7634.754]);
BIG = 1000;
SMALL = 1e-3;
problem = struct();
problem.f = list(normdiff2,t,x_exp);
problem.x0 = k0;
problem.x_lower = [SMALL*ones(16,1);100];
problem.x_upper = [BIG,BIG,BIG,BIG,BIG,BIG,3,3,3,3,3,3,3,3,3,3,10000]';
problem.int_cb = callback;
problem.params = struct("max_iter",200);
//
k = ipopt(problem)
Here is a plot of the results after 100 iterations (you can change the value in the ipopt options). However, don't expect normal termination as
it is almost sure that your parameter set is not identifiable
finite difference gradient with ODEs is very innacurate.
Hope it will help you a little bit...
I've researched previous similar questions, but couldn't find any applicable leads:
I have a dataframe, called "df" which is roughly structured as follows:
Income Income_Quantile Score_1 Score_2 Score_3
0 100000 5 75 75 100
1 97500 5 80 76 94
2 80000 5 79 99 83
3 79000 5 88 78 91
4 70000 4 55 77 80
5 66348 4 65 63 57
6 67931 4 60 65 57
7 69232 4 65 59 62
8 67948 4 64 64 60
9 50000 3 66 50 60
10 49593 3 58 51 50
11 49588 3 58 54 50
12 48995 3 59 59 60
13 35000 2 61 50 53
14 30000 2 66 35 77
15 12000 1 22 60 30
16 10000 1 15 45 12
Using the "Income_Quantile" column and the following "for-loop", I divided the dataframe into a list of 5 subset dataframes (which each contain observations from the same income quantile):
dfs = []
for level in df.Income_Quantile.unique():
df_temp = df.loc[df.Income_Quantile == level]
dfs.append(df_temp)
Now, I would like to apply the following function for calculating the spearman correlation, p-value and t-statistic to the dataframe (fyi: scipy.stats functions are used in the main function):
def create_list_of_scores(df):
df_result = pd.DataFrame(columns=cols)
df_result.loc['t-statistic'] = [ttest_ind(df['Income'], df[x])[0] for x in cols]
df_result.loc['p-value'] = [ttest_ind(df['Income'], df[x])[1] for x in cols]
df_result.loc['correlation'] = [spearmanr(df['Income'], df[x])[1] for x in cols]
return df_result
The functions that "create_list_of_scores" uses, i.e. "ttest_ind" and "ttest_ind", can be accessed from scipy.stats as follows:
from scipy.stats import ttest_ind
from scipy.stats import spearmanr
I tested the function on one subset of the dataframe:
data = dfs[1]
result = create_list_of_scores(data)
It works as expected.
However, when it comes to applying the function to the entire list of dataframes, "dfs", a lot of issues arise. If I apply it to the list of dataframes as follows:
result = pd.concat([create_list_of_scores(d) for d in dfs], axis=1)
I get the output as the columns "Score_1, Score_2, and Score_3" x 5.
I would like to:
Have just three columns "Score_1, Score_2, and Score_3".
Index the output using the t-statistic, p-value and correlations as the first level index, and; the "Income_Quantile" as the second level index.
Here is what I have in mind:
Score_1 Score_2 Score_3
t-statistic 1
2
3
4
5
p-value 1
2
3
4
5
correlation 1
2
3
4
5
Any idea on how I can merge the output of my function as requested?
I think better is use GroupBy.apply:
cols = ['Score_1','Score_2','Score_3']
def create_list_of_scores(df):
df_result = pd.DataFrame(columns=cols)
df_result.loc['t-statistic'] = [ttest_ind(df['Income'], df[x])[0] for x in cols]
df_result.loc['p-value'] = [ttest_ind(df['Income'], df[x])[1] for x in cols]
df_result.loc['correlation'] = [spearmanr(df['Income'], df[x])[1] for x in cols]
return df_result
df = df.groupby('Income_Quantile').apply(create_list_of_scores).swaplevel(0,1).sort_index()
print (df)
Score_1 Score_2 Score_3
Income_Quantile
correlation 1 NaN NaN NaN
2 NaN NaN NaN
3 6.837722e-01 0.000000e+00 1.000000e+00
4 4.337662e-01 6.238377e-01 4.818230e-03
5 2.000000e-01 2.000000e-01 2.000000e-01
p-value 1 8.190692e-03 8.241377e-03 8.194933e-03
2 5.887943e-03 5.880440e-03 5.888611e-03
3 3.606128e-13 3.603267e-13 3.604996e-13
4 5.584822e-14 5.587619e-14 5.586583e-14
5 3.861801e-06 3.862192e-06 3.864736e-06
t-statistic 1 1.098143e+01 1.094719e+01 1.097856e+01
2 1.297459e+01 1.298294e+01 1.297385e+01
3 2.391611e+02 2.391927e+02 2.391736e+02
4 1.090548e+02 1.090479e+02 1.090505e+02
5 1.594605e+01 1.594577e+01 1.594399e+01
Tensorflow Graph Transforms page https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/graph_transforms/README.md shows how to use strip_unused_nodes.
But how to know the right values of X and Y in strip_unused_nodes(type=X, shape="y0,y1,y3,3") for my model?
Output of summarize_graph on my MobileNetV2 model :
Found 1 possible inputs: (name=image_tensor, type=uint8(4), shape=[?,?,?,3])
No variables spotted.
Found 4 possible outputs: (name=detection_boxes, op=Identity) (name=detection_scores, op=Identity) (name=detection_classes, op=Identity) (name=num_detections, op=Identity)
Found 3457096 (3.46M) const parameters, 0 (0) variable parameters, and 623 control_edges
Op types used: 1707 Const, 525 Identity, 277 Mul, 194 Add, 170 Reshape, 147 GatherV2, 133 Sub, 117 Minimum, 98 Slice, 92 Maximum, 77 ConcatV2, 77 Cast, 64 Rsqrt, 60 StridedSlice, 59 Relu6, 55 Conv2D, 54 Pack, 52 Greater, 49 Shape, 46 Split, 46 Where, 45 ExpandDims, 40 Fill, 37 Tile, 33 RealDiv, 33 DepthwiseConv2dNative, 30 Range, 29 Switch, 27 Unpack, 26 Enter, 25 Squeeze, 25 ZerosLike, 23 NonMaxSuppressionV2, 14 Merge, 12 BiasAdd, 12 FusedBatchNorm, 11 TensorArrayV3, 8 NextIteration, 6 TensorArrayWriteV3, 6 TensorArraySizeV3, 6 Sqrt, 6 Exit, 6 TensorArrayGatherV3, 5 TensorArrayScatterV3, 5 TensorArrayReadV3, 3 Rank, 3 Equal, 3 Transpose, 3 Assert, 2 Exp, 2 Less, 2 LoopCond, 1 All, 1 TopKV2, 1 Size, 1 Sigmoid, 1 ResizeBilinear, 1 Placeholder
To use with tensorflow/tools/benchmark:benchmark_model try these arguments:
bazel run tensorflow/tools/benchmark:benchmark_model -- --graph=/home/ubuntu/model-optimization/frozen_inference_graph.pb --show_flops --input_layer=image_tensor --input_layer_type=uint8 --input_layer_shape=-1,-1,-1,3 --output_layer=detection_boxes,detection_scores,detection_classes,num_detections
I believe you should copy the input layer dims, you can find in the .ascii file of your model
I need to create spline programmatically. I've made something like:
0
SECTION
2
HEADER
9
$ACADVER
1
AC1006
0
ENDSEC
0
SECTION
2
TABLES
0
TABLE
2
LAYER
0
LAYER
2
shape
70
64
62
250
6
CONTINUOUS
0
LAYER
2
holes
70
64
62
250
6
CONTINUOUS
0
ENDTAB
0
ENDSEC
0
SECTION
2
ENTITIES
0
SPLINE
8
shape
100
AcDbSpline
210
0
220
0
230
1
70
4
71
3
72
11
73
4
74
4
42
0.0000001
43
0.0000001
44
0.0000000001
40
0
40
0
40
0
40
0
40
1
40
1
40
1
40
2
40
2
40
2
40
2
10
0
20
0
30
0
10
100
20
50
30
0
10
40
20
40
30
0
10
15
20
23
30
0
11
0
21
0
31
0
11
200
21
200
31
0
11
80
21
80
31
0
11
432
21
234
31
0
0
ENDSEC
0
EOF
When I'm trying to open it in Autodesk TrueView, I'm getting an error:
Undefined group code 210 for object on line 54.
Invalid or incomplete DXF input -- drawing discarded.
Where is the error? When I'm copying just SPLINE section to the DXF generated by AI everything works fine. So I think I need to add something in the header section or something.
This file is DXF version AC1006 which is older than DXF R12. The SPLINE entity
requires at least DXF version AC1012 DXF R13/R14. But with DXF version AC1012
the tag structure of DXF files is changed (OBJECTS and CLASSES sections, SubClassMarkers ...), so just editing the DXF version
does not work.
See also: http://ezdxf.readthedocs.io/en/latest/dxfinternals/filestructure.html#minimal-dxf-content
Also the SPLINE entity seems to be invalid, it has no handle (5) and no owner
tag (330), and the whole AcDbEntity subclass is missing.
Your spline is of degree 3 with 11 knots (0, 0,0,0,1,1,1,2,2,2,2) and 4 control points ( (0,0), (100,50),(40,40),(15,23) ). This might be the problem culprit. You should either have 4 control points and 8 knots or 7 control points and 11 knots.
You may need to assign a handle to the SPLINE, since you're specifying $ACADVER = AC1018 = AutoCAD 2004 where item handles are required.
Try adding a 5-code pair right before the layer designation, like so, where AAAA is a unique hex-encoded handle:
0
SPLINE
5 <-- add these two lines
AAAA <--
8
shape
100
AcDbSpline
I'm playing around with Seaborn and Matplotlib and I trying to find the best type of graph to show the correlation between fare values and chance of survival from the titanic dataset.
The Titanic fare column has a lot of different values ranging from 1 to 500 and some of the values are repeated often.
Here is a sample of value_counts:
titanic.fare.value_counts()
8.0500 43
13.0000 42
7.8958 38
7.7500 34
26.0000 31
10.5000 24
7.9250 18
7.7750 16
0.0000 15
7.2292 15
26.5500 15
8.6625 13
7.8542 13
7.2500 13
7.2250 12
16.1000 9
9.5000 9
15.5000 8
24.1500 8
14.5000 7
7.0500 7
52.0000 7
31.2750 7
56.4958 7
69.5500 7
14.4542 7
30.0000 6
39.6875 6
46.9000 6
21.0000 6
.....
91.0792 2
106.4250 2
164.8667 2
Survival column on the other hand has only two values :
>>> titanic.survived.head(10)
271 1
597 0
302 0
633 0
277 0
413 0
674 0
263 0
466 0
A histogram would only show the frequency of fares in certain ranges.
For a scatter plot I would need two variables; having "survived" which has only two values would make for a strange variable.
Is there a way to show the rise of survivability as fare increases clearly through a line graph?
I know there is a correlation as If I sort fare values in ascending order (000-500).
Then do:
>>> titanic.head(50).survived.sum()
5
>>>titanic.tail(50).survived.sum()
37
I see a correlation.
Thanks.
This is what I did to show the correlation between the fare values and the chance of survival:
First, I created a new column Fare Groups, converting fare values to groups of fare ranges, using cut().
df['Fare Groups'] = pd.cut(df.Fare, [0,50,100,150,200,550])
Next, I created a pivot_table().
piv_fare = df.pivot_table(index='Fare Groups', columns='Survived', values = 'Fare', aggfunc='count')
Output:
Survived 0 1
Fare Groups
(0, 50] 484 232
(50, 100] 37 70
(100, 150] 5 19
(150, 200] 3 6
(200, 550] 6 14
Plot:
piv_fare.plot(kind='bar')
It seems, those who had the cheapest tickets (0 to 50) had the lowest chance of survival. In fact, (0 to 50) is the only fare range where the chance to die is higher than the chance to survive. Not just higher, but significantly higher.