I have a small trouble with the performance of the exception handling.
I'm building... aaaaa... soooo...hmmm...a graph db? yeah something like this... with vb.net.
I'm building the parser, for handling the basic functions of the db(?) What happen? The user makes a research, the program tab them, and allow to make some computation . Normally string handling, but, to give a complete instrument, I implement also mathematical functions.
There is no way to know developing-time math kind of value he is going to insert. And especially, there is no way to know if each row of the column that comes out the traversal has the same data type. It is a schema-free db(?).
So what happen...I have to implement the error handling.
For i As Integer = 0 To pR.r.wsRowCount - 1
Try
pR.r.wsItem(cix, i) = Convert.ToDouble(pR.r.wsItem(ca.ix, i)) * Convert.ToDouble(pR.r.wsItem(cb.ix, i))
Catch
pR.r.wsItem(cix, i) = String.Empty
End Try
Next
Something like this...
(psss.. Convert.toDouble() is the most performant function between cDbl(), Double.Parse() etc).
But when I run it, if all the rows (benchmark around 2000 rows), are really numeric... few milliseconds, but if some rows are not numeric, it require 3 or 4 seconds. And worst scenario, if the user should do a mathematical operation in a string column... goodbye, meanwhile is better go to take a coffee.
So my computer may be is not the most advanced in the world, but are you aware if there is any way to avoid this incredible delay in the error handling? I would remember that this is the parser of a query and I would avoid to make it heavy.
You are using Exceptions to fix a problem when your code encounter a value that cannot be converted to a double. This is called driving your code using exceptions and it is a bad practice. Catching exceptions is never a good fix for this scenario.
In particular when there is a clear way to avoid them
For i As Integer = 0 To pR.r.wsRowCount - 1
Dim d1 As Double
Dim d2 As Double
if Double.TryParse(pR.r.wsItem(ca.ix, i), d1) Then
if Double.TryParse(pR.r.wsItem(cb.ix, i), d2 Then
pR.r.wsItem(cix, i) = (d1 * d2).ToString()
else
pR.r.wsItem(cix, i) = String.Empty
End If
Else
pR.r.wsItem(cix, i) = String.Empty
End If
Next
Convert.ToDouble cannot handle the fact that a string is not a valid numeric value and if it encounter this case it throws an exception. Throwing an exception is costly in terms of perfomance in particular for gathering the information about the call stack.
Instead, if there is the possibility that your input contains values that cannot be converted to double, then Double.TryParse is the correct way to proceed also if your benchmarks show a little difference in performance. Double.TryParse doesn't raise an exception if the string passed cannot be converted to a double, but return false. In this way the costly exception is avoided and you have a more predictable time of execution
Said that, you should really reevaluate your approach to store every kind of data in some string structure. The need of constant conversion between a string and the intended data type could be a real bottleneck here.
Related
I'm struggling with the Excel Operators I have spent quite some time searching I feel as if I am close I'm trying to code this If statement if the value does not begin with 722* and does not equal PI then execute code immediately below else execute code under the else statement.
If Not CStr(Cells(r, cC)) = "722*" And Not CStr(Cells(r, cS)) = "PI" Then
The easiest way would be to use Left
If Not Left(CStr(Cells(r, cC),3) = "722"
If you want to use wildcards, use Like:
If Not Cells(r, cC)) Like "722*" Then
Comparing them with = treats * simply as a character.
Edit2: apparently I was wrong about the computational effort of Like. Like seems to be the faster method (in this case), especially if there is no match (lazy evaluation I guess). Obviously the total cpu and memory usage is so small in both cases that it's up to your preference which one to choose.
This is what I found:
s Like "722*" is faster than Left(s,3) = "722"
s Like "*722" is slower than Right(s,3) = "722"
s Like "*722*" is slower than Instr(1,s,"722")>0
The magnitude of the difference varies depending on string length and if there is a match or not (e.g. s Like "722*" is very fast if s doesn't start with 7 and Left and Right don't suffer as much as Like if you replace "722" with something longer)
I've created an interprter for a simple language. It is AST based (to be more exact, an irregular heterogeneous AST) with visitors executing and evaluating nodes. However I've noticed that it is extremely slow compared to "real" interpreters. For testing I've ran this code:
i = 3
j = 3
has = false
while i < 10000
j = 3
has = false
while j <= i / 2
if i % j == 0 then
has = true
end
j = j+2
end
if has == false then
puts i
end
i = i+2
end
In both ruby and my interpreter (just finding primes primitively). Ruby finished under 0.63 second, and my interpreter was over 15 seconds.
I develop the interpreter in C++ and in Visual Studio, so I've used the profiler to see what takes the most time: the evaluation methods.
50% of the execution time was to call the abstract evaluation method, which then casts the passed expression and calls the proper eval method. Something like this:
Value * eval (Exp * exp)
{
switch (exp->type)
{
case EXP_ADDITION:
eval ((AdditionExp*) exp);
break;
...
}
}
I could put the eval methods into the Exp nodes themselves, but I want to keep the nodes clean (Terence Parr saied something about reusability in his book).
Also at evaluation I always reconstruct the Value object, which stores the result of the evaluated expression. Actually Value is abstract, and it has derived value classes for different types (That's why I work with pointers, to avoid object slicing at returning). I think this could be another reason of slowness.
How could I make my interpreter as optimized as possible? Should I create bytecodes out of the AST and then interpret bytecodes instead? (As far as I know, they could be much faster)
Here is the source if it helps understanding my problem: src
Note: I haven't done any error handling yet, so an illegal statement or an error will simply freeze the program. (Also sorry for the stupid "error messages" :))
The syntax is pretty simple, the currently executed file is in OTZ1core/testfiles/test.txt (which is the prime finder).
I appreciate any help I can get, I'm really beginner at compilers and interpreters.
One possibility for a speed-up would be to use a function table instead of the switch with dynamic retyping. Your call to the typed-eval is going through at least one, and possibly several, levels of indirection. If you distinguish the typed functions instead by name and give them identical signatures, then pointers to the various functions can be packed into an array and indexed by the type member.
value (*evaltab[])(Exp *) = { // the order of functions must match
Exp_Add, // the order type values
//...
};
Then the whole switch becomes:
evaltab[exp->type](exp);
1 indirection, 1 function call. Fast.
I posted a code example in previous question and here is critical part:
Try
first = _string.IndexOf(_firstchar) + 1
second = _string.IndexOf(_secondchar)
If first >= 1 And second >= 0 Then
retval = _string.Substring(first, second - first)
End If
Catch ex As Exception
End Try
One of experienced VB users say that such usage of Try/Catch is "evil".
Why is such usage "evil" and what is minimally enough to get program to continue when we don't want to bother with error?
An empty Catch block is evil since it can hide exceptions that you should handle. The Try-Catch is redundant here, you should use the overload with a start index for the second :
first = _string.IndexOf(_firstchar)
If first >= 0 Then
first += 1
second = _string.IndexOf(_secondchar, first)
If second >= 0 Then
retval = _string.Substring(first, second - first)
End If
End If
Note that you should use AndAlso instead of And.
The reason that is considered bad practice is because the exception you are catching there is not really handled in any way so it goes unnoticed. That in turn can cause other problems further down the program flow and those would be heard to gave back to the root cause, or even if traced, fixing them might be complicated by the previous decision to mask the error condition.
That said, there may be rare circumstances where it would be appropriate to swallow an exception like that. But if you do make that decision make sure to document the fact and state the reason you decided to ignore the error at that point.
For instance, it might be appropriate to ignore an exception that is triggered by a logging function. Since the purpose of the logger is to log the error, if for some reason the logging fails, there is probably not much else to do with that exception..
I am trying to compute numerically the solutions for a system of many equations and variables (100+). I tried so far three things:
I now that the vector of p(i) (which contains most of the endogenous variables) is decreasing. Thus I gave simply some starting points, and then was increasing(decreasing) my guess when I saw that the specific p was too low(high). Of course this was always conditional on the other being fixed which is not the case. This should eventually work, but it is neither efficient, nor obvious that I reach a solution in finite time. It worked when reducing the system to 4-6 variables though.
I could create 100+ loops around each other and use bisection for each loop. This would eventually lead me to the solution, but take ages both to program (as I have no idea how to create n loops around each other without actually having to write the loops - which is also bad as I would like to increase/decrease the amount of variables easily) and to execute.
I was trying fminsearch, but as expected for that wast amount of variables - no way!
I would appreciate any ideas... Here is the code (this one the fminsearch I tried):
This is the run file:
clear all
clc
% parameter
z=1.2;
w=20;
lam=0.7;
tau=1;
N=1000;
t_min=1;
t_max=4;
M=6;
a_min=0.6;
a_max=0.8;
t=zeros(1,N);
alp=zeros(1,M);
p=zeros(1,M);
p_min=2;
p_max=1;
for i=1:N
t(i)= t_min + (i-1)*(t_max - t_min)/(N-1);
end
for i=1:M
alp(i)= a_min + (i-1)*(a_max - a_min)/(M-1);
p(i)= p_min + (i-1)*(p_max - p_min)/(M-1);
end
fun=#(p) david(p ,z,w,lam,tau,N,M,t,alp);
p0=p;
fminsearch(fun,p0)
And this is the program-file:
function crit=david(p, z,w,lam,tau,N,M,t,alp)
X = zeros(M,N);
pi = zeros(M,N);
C = zeros(1,N);
Xa=zeros(1,N);
Z=zeros(1,M);
rl=0.01;
rh=1.99;
EXD=140;
while (abs(EXD)>100)
r1=rl + 0.5*(rh-rl);
for i=1:M
for j=1:N
X(i,j)=min(w*(1+lam), (alp(i) * p(i) / r1)^(1/(1-alp(i))) * t(j)^((z-alp(i))/(1-alp(i))));
pi(i,j)=p(i) * t(j)^(z-alp(i)) * X(i,j)^(alp(i)) - r1*X(i,j);
end
end
[C,I] = max(pi);
Xa(1)=X(I(1),1);
for j=2:N
Xa(j)=X(I(j),j);
end
EXD=sum(Xa)- N*w;
if (abs(EXD)>100 && EXD>0)
rl=r1;
elseif (abs(EXD)>100 && EXD<0)
rh=r1;
end
end
Ya=zeros(M,N);
for j=1:N
Ya(I(j),j)=t(j)^(z-alp(I(j))) * X(I(j),j)^(alp(I(j)));
end
Yi=sum(Ya,2);
if (Yi(1)==0)
Z(1)=-50;
end
for j=2:M
if (Yi(j)==0)
Z(j)=-50;
else
Z(j)=(p(1)/p(j))^tau - Yi(j)/Yi(1);
end
end
zz=sum(abs(Z))
crit=(sum(abs(Z)));
First of all my recommendation: use your brain.
What do you know about the function, can you use a gradient approach, linearize the problem, or perhaps fix most of the variables? If not, think twice before you decide that you are really interested in all 100 variables and perhaps simplify the problem.
Now, if that is not possible read this:
If you found a way to quickly get a local optimum, you could simply wrap a loop around it to try different starting points and hope you will find a good optimum.
If you really need to make lots of loops (and a variable amount) I suppose it can be done with recursion, but it is not easily explained.
If you just quickly want to make a fixed number of loops inside each other this can easily be done in excel (hint: loop variables can be called t1,t2 ... )
If you really need to evaluate a function at a lot of points, probably creating all the points first using ndgrid and then evaluating them all at once is preferable. (Needless to say this will not be a nice solution for 100 nontrivial variables)
I’ve got a massive memory leak in my program. This is the first time I’ve used IronPython in a tight loop, so I’m wondering if this could be the cause.
For Each column In m_Columns
Dim rawValue As String
rawValue = line.Substring(column.FileColumnIndex.Value - 1, column.FileColumnEndIndex.Value - column.FileColumnIndex.Value + 1)
If column.IncludeColumnFunction <> "" Then
Dim scope = m_ScriptEngine.CreateScope
scope.SetVariable("Line", line)
scope.SetVariable("Row", targetRow)
If Not CBool(m_ScriptEngine.Execute(column.IncludeColumnFunction, scope)) Then Continue For 'skip this column
End If
targetRow(column.DatabaseColumnName) = column.ParseValue(rawValue, targetRow)
Next
The string named column.IncludeColumnFunction never changes for a given column. It is usually something simple like “Row['Foo'] == 'Bar'”.
Can I/should I be caching the compiled function?
Should I be destroying the scope variable in some way when I’m done with it?
Nothing in particular stands out in the code sample that you put up there. I would say it's unlikely that this particular piece of code is causing the issue (although more context is needed to be definitive).
With memory leaks you'll track down the problem much faster by directly investigating the problem vs. digging through code. Direct investigations will often quickly tell you what is leaking and once you know what is leaking then you can start investigating code which manages that type of object.
There are lots of tools and articles out there to help track down memory issues in .Net. Here is a particularly good one.
http://blogs.msdn.com/b/tess/archive/2008/02/15/net-debugging-demos-lab-3-memory.aspx?wa=wsignin1.0