VB.Net slow down on array size increase - vb.net

Question

In my array processing intensive code, the rate of processing items slows dramatically after around 12,000 items with 48,000 operating at <50% of the speed.
The code undertakes maths on the array items and repeats that maths say 1000 times. At 12,000 items in the array I can process at a rate of 40,000 per second at 48,000 items in the array I can only process at a rate of 15,000 per second.
The code reads the contents of an Excel sheet then hands off to a DLL file with all the arrays passed to the DLL, which then processes them and hands the adjusted arrays back to Excel. Once running the DLL code, there is no interfacing with Excel. No "objects" (as in an Excel object) are created inside the VB.Net.
This was VBA code that I've transferred to VB.Net (first time using it) so bound to be something dumb I haven't realised.
Just to confuse matters...
In the Net code you'll see I've used % for integer values, which I changed from the original &, as 32-bit versions of Excel threw errors in passing their & types to the DLL which was written on a 64-bit platform, if that makes sense. Aware of the change in data types from VBA to VB.Net, that is & in vba is I think the same as % in VB.Net.
Advanced apology....
I am an amateur coder and this is a project which went from a 'could I?' project and grew and grew. It does work and is fast, I just noticed this performance drop off.
Tried switching off the calling of the ANNIE_TEST function which occurs every 50 iterations through the main loop. Makes no difference in performance.
Tried changing arrays to ByVal instead of ByRef, even though I read VB.Net ignores that for arrays - no difference.
Wondering if this is garbage collection (not that I really understand that) or perhaps the use of % integers instead of & integers.
<ComClass(clsAnnie.ClassId, clsAnnie.InterfaceId, clsAnnie.EventsId)>
Public Class clsAnnie
Public Const ClassId As String = "00E45698-E5C1-4F2C-8828-3F7404EF83A2"
Public Const InterfaceId As String = "2A1D9402-78DA-41F0-8DE3-1CD573AAF72C"
Public Const EventsId As String = "346E24CC-C9E1-4915-B254-BA80E6E310C4"
Public Sub New()
MyBase.New
End Sub
'This routine is called repeatedly from the entry point
'sub-routine Public Sub Train_NNT
Private Sub Calculate_NNT(ByVal Num_Layers%, ByVal Pattern%, ByVal Num_Hidden%, ByVal Num_Inputs%, ByRef Hidden_Neuron_Value As Double(,,), ByRef Prediction_Output As Double(,), ByRef Pattern_Error As Double(), ByRef Training_Input As Double(,), ByRef Training_Output As Double(), ByRef Input_Weights As Double(,), ByRef Output_Weights As Double(,), ByRef Inter_Weights As Double(,,), ByVal drop_out As Boolean, ByRef doi As Integer(), ByRef dol As Integer(,), ByVal activation_function%)
On Error GoTo exity
Dim alldropped As Boolean, n#, a#, c#, m%, s%, Lay%, i%, j%
'The main code is removed to keep the post short, but all it does is mathematically
'loop through the arrays and undertake maths on the values before updating one
'particular array (Training_Output) and passing that back to the main routine.
'None of the other arrays or variables have their values changed.
'Last few lines as follows for context as follows....
Prediction_Output(Pattern%, 1) = 0
For m% = 1 To Num_Hidden% : Prediction_Output(Pattern%, 1) = Prediction_Output(Pattern%, 1) + (Hidden_Neuron_Value(Pattern%, Num_Layers%, m%) * Output_Weights(m%, 1)) : Next
Pattern_Error(Pattern%) = Prediction_Output(Pattern%, 1) - Training_Output(Pattern%)
exity:
End Sub
'This is the main routine that is called from Excel VBA. It's called only once.
Public Sub Train_NNT(ByRef doi As Integer(), ByRef dol As Integer(,), ByRef OF_Array As Double(,), ByRef Cyc%, ByRef Hidden_Neuron_Value As Double(,,), ByRef Pattern_Error As Double(), ByRef Inter_Weights As Double(,,), ByRef Prediction_Output As Double(,), ByRef Training_Input As Double(,), ByRef Training_Output As Double(), ByRef Training_Input_Test As Double(,), ByRef Training_Output_Test As Double(), ByRef Input_Weights As Double(,), ByRef Output_Weights As Double(,), ByVal activation_function%, ByVal Num_Patterns%, ByVal Cycles%, ByVal Num_Layers%, ByVal Num_Hidden%, ByVal Num_Inputs%, ByVal Num_Patterns_Test%, ByVal Learn_Rate#, ByVal Maximum_Data_Value As Double, ByVal Minimum_Data_Value As Double, ByVal DOLLR As Boolean, ByVal Overfitting_CF As Boolean, ByVal Lregression As Boolean, ByVal cosine_annealing As Boolean, ByVal ADAMW As Boolean, ByVal Nesterov As Boolean, ByVal drop_out As Boolean, ByVal random_sampling As Boolean, ByVal animations As Boolean)
On Error GoTo exit_train
'These declarations shown in case it's something to do with garbage collection?
Dim Min_CET#, Best_Error#, Best_Error_Test#, Current_Error#, Current_Error_Test#, Learn_Rate_Original#, Learn_Rate_Min#, Learn_Mult#, Learn_Rate_Const#, lamda#
Dim beta_1#, beta_1_1#, beta_2#, beta_2_2#, mom_1#, mom_2#, mom_3#, damp#, cos_tmp#, beta_1_2#, tmp#, n#, P_Pat_Error#, Learn_Rate_Inv#, beta_sqr#, epsilon#
Dim Pat_Error#, Pat_Error_M#, tmp_5#, dw#, derivative#, tmp_1#, MaDV#, MiDV#
Dim Picture_Count As Integer, Picture_VE As Integer
Dim error_counter%, Snaps%, Pattern%, i%, s%, m%, o%, Pat_Test%, Pat%, q%, Lay%, j%
Dim Error_Array(0 To 100) As Double, HNV(0 To Num_Layers%, 0 To Num_Hidden%) As Double, neuron_error(0 To Num_Layers%, 0 To Num_Hidden%) As Double
Dim vdw_o() As Double, sdw_o() As Double, vdw_i() As Double, sdw_i() As Double, Output_Momentum() As Double, Input_Momentum(,) As Double, Inter_Momentum(,,) As Double
ReDim vdw_o(0 To (Num_Hidden% + 1) * (Num_Inputs% + Num_Layers% + 2)), sdw_o(0 To (Num_Hidden% + 1) * (Num_Inputs% + Num_Layers% + 2)), vdw_i(0 To (Num_Hidden% + 1) * (Num_Inputs% + Num_Layers% + 2)), sdw_i(0 To (Num_Hidden% + 1) * (Num_Inputs% + Num_Layers% + 2))
ReDim Output_Momentum(0 To Num_Hidden%), Input_Momentum(0 To (Num_Inputs% + 1), 0 To Num_Hidden%), Inter_Momentum(0 To Num_Layers%, 0 To Num_Hidden%, 0 To Num_Hidden%)
'Here we enter a loop which processes the arrays
For Cyc% = 1 To Cycles%
'Code removed here as just altering some variables and performing maths on them.
'I call the routine above...
Call Calculate_NNT(Num_Layers%, Pattern%, Num_Hidden%, Num_Inputs%, Hidden_Neuron_Value, Prediction_Output, Pattern_Error, Training_Input, Training_Output, Input_Weights, Output_Weights, Inter_Weights, drop_out, doi, dol, activation_function%)
'Lot of code removed, uses the answer from the routine above to process and alter
'a number of arrays. The array passed back (or is used) is
'Training_Output
'Every 50 cycles I need to call a function called ANNIE_TEST
'which independently verifies whether the maths hasn't exceeded
'certain thresholds, which if it has then stops the whole process
'and hands back to Excel.
'Code below just for context
If Cyc% Mod 50 = 0 Then
If Overfitting_CF Then
Current_Error_Test# = 0
For Pat_Test% = 1 To Num_Patterns_Test% : Current_Error_Test# = Current_Error_Test# + (ANNIE_TEST(Pat_Test%, Num_Hidden%, Num_Inputs%, Num_Layers%, activation_function%, Output_Weights, Inter_Weights, Training_Input_Test, Input_Weights, HNV) - Training_Output_Test(Pat_Test%)) ^ 2 : Next
Current_Error_Test# = Math.Sqrt(Current_Error_Test# / Num_Patterns_Test%)
OF_Array(q%, 1) = Cyc% : OF_Array(q%, 2) = Current_Error_Test# : q% = q% + 1
Min_CET# = 0
For o% = 1 To error_counter% : Min_CET# = Min_CET# + Error_Array(o%) : Next
Min_CET# = Min_CET# / error_counter%
If Current_Error_Test# <= Min_CET# Then
For o% = error_counter% To 2 Step -1 : Error_Array(o%) = Error_Array(o% - 1) : Next
Error_Array(1) = Current_Error_Test#
Else
Exit For
End If
End If
End If
Next 'cycles
exit_train:
End Sub
Private Function ANNIE_TEST(ByVal Pat_Test%, ByVal Num_Hidden%, ByVal Num_Inputs%, ByVal Num_Layers%, ByVal activation_function%, ByRef Output_Weights As Double(,), ByRef Inter_Weights As Double(,,), ByRef Training_Input_Test As Double(,), ByRef Input_Weights As Double(,), ByRef HNV As Double(,))
Dim m%, j%, s%, Lay%
Dim a#, c#, n#, prediction#
'Code removed as it just loops through arrays and undertakes maths on them.
'None of the arrays have values altered, just used
prediction# = 0
For j% = 1 To Num_Hidden% : prediction# = prediction# + (HNV(Num_Layers%, j%) * Output_Weights(j%, 1)) : Next
ANNIE_TEST = prediction#
End Function
End Class

Answer 1

OK turns out nothing to do with VB.Net and everything to do with how the arrays were handled after they arrived back in Excel, and it was really dumb. They ended up in a function which needed to load the arrays in, which had a constant so it could tell on a successive run of the function that the arrays had been loaded. I forgot to declare that constant as a public boolean, and therefore Excel just saw "null" instead of true/false.
I want to sincerely thank all of you for helping and especially TnTinMn, for making me think (like I should have done from the start) about where the timings were being taken.
Going from VBA to VB.Net gave a 7 to 10x speed increase, the fix to this has given me a further 7x speed increase on top.