vb.net Searching for specific lines within large .log files - vb.net

Question

I'm reading from 1-5mb log files outputting to a textview and also searching for specific lines outputting to another textview. Currently it takes about a minute for just a 1mb file. Does anyone know any faster methods of searching through lines or strings other than the method I'm using?
Imports EnterpriseDT.Net.Ftp
Public Class Form1
Private Sub SettingsToolStripMenuItem_Click(sender As System.Object, e As System.EventArgs) Handles SettingsToolStripMenuItem.Click
End Sub
Private Sub Button2_Click(sender As System.Object, e As System.EventArgs) Handles Button2.Click
Dim sw As New Stopwatch
Dim FullLine As String = ""
Dim ScriptLine As String = ""
sw.Start()
Dim ll As New Queue(Of String)
Dim i As String = ""
Using TestFile As New IO.StreamReader("c:\test.txt", System.Text.Encoding.Default, False, 4096)
Using OutFile As New IO.StreamWriter("c:\SBOutFile.txt", False, System.Text.Encoding.Default, 4096)
While TestFile.EndOfStream = False
i = TestFile.ReadLine
If i.Contains(".sqf") And i.Contains("handleGear.sqf") = False Then
ScriptLine = ScriptLine & i & vbNewLine & vbNewLine
FullLine = FullLine & i & vbNewLine & vbNewLine
Else
FullLine = FullLine & i & vbNewLine & vbNewLine
End If
End While
End Using
End Using
sw.Stop()
TextBox1.Text = FullLine
TextBox2.Text = ScriptLine
RichTextBox1.AppendText(String.Format("Run_Queue took {0} Milliseconds." & Environment.NewLine, sw.ElapsedMilliseconds))
End Sub
Private Sub Button1_Click_1(sender As System.Object, e As System.EventArgs) Handles Button1.Click
Try
'connect to ftp server
Dim ftp As New FTPConnection
ftp.ServerAddress = "-"
ftp.ServerPort = "-"
ftp.UserName = "-"
ftp.Password = "-"
ftp.Connect()
ftp.ChangeWorkingDirectory("-")
ftp.TransferType = FTPTransferType.BINARY
'download a file
ftp.DownloadFile("c:\test.txt", "scripts.log")
'ftp.RenameFile("scripts.log", "scripts_test.log")
'close the connection
ftp.Close()
Catch ex As Exception
MessageBox.Show(ex.Message.ToString())
End Try
End Sub
Private Sub Button3_Click(sender As System.Object, e As System.EventArgs) Handles Button3.Click
End Sub
End Class

Answer 1

Given the heavy reading and concatenating you are doing as you're reading, I suspect its part of your time/performance issue. I would probably consider changing the declarations for ScriptLine and FullLine from String type to a StringBuilder, because Strings are technically immutable. That means each concatenation really turns out to be teardown of the previous object, and the creation of a new one in its place. StringBuilders are designed specifically for heavy concatenation scenarios. When the looping is finished, you can convert it back to a String.
Also, a compiled Regular Expression might search faster than String.Contains. Your regular expression string would be something like "(?!handleGear).sqf", meaning "find any sequence of zero or more characters other than "handleGear" in front of the string ".sqf".
I haven't had a chance to test that expression, so it is offered with that caveat. If I get a chance to throw together a test, I'll be glad to amend and let you know.
Good luck!

Answer 2

I just wanted to post what I came up with in the end. It was a very large improvement to what I had.
'Read file
Dim sw As New Stopwatch
Dim FullLine As String = ""
Dim ScriptLine As String = ""
sw.Start()
Dim ll As New Queue(Of String)
Dim i As String = ""
Dim builder As New StringBuilder
Using TestFile As New IO.StreamReader("c:\test.txt", System.Text.Encoding.Default, False, 4096)
builder.AppendLine("Started at: " & DateTime.Now.ToLongTimeString().ToString)
RichTextBox1.AppendText(Now.ToShortTimeString & " Reading Log File Started" & vbNewLine)
RichTextBox1.SelectionStart = RichTextBox1.TextLength
Dim rowCount As Integer = 0
Do Until TestFile.EndOfStream
ScriptLine = TestFile.ReadLine
ScriptLine = LCase(ScriptLine)
If InStr(ScriptLine, ".sqf") > 0 And InStr(ScriptLine, "handlegear.sqf") < 1 Then 'And InStr(ScriptLine, "createmarkerlocal.sqf") < 1 And InStr(ScriptLine, "setmarkerposlocal.sqf") < 1
builder.AppendLine(ScriptLine)
builder.AppendLine()
End If
rowCount = rowCount + 1
Loop
builder.AppendLine(Now.ToShortTimeString & "==== Searched " & rowCount & " rows" & vbNewLine)
builder.AppendLine(Now.ToShortTimeString & " Finished" & vbNewLine)
End Using
sw.Stop()
RichTextBox2.AppendText(builder.ToString & vbNewLine)
RichTextBox1.AppendText(Now.ToShortTimeString & String.Format(" Run_Queue took {0} Milliseconds." & Environment.NewLine, sw.ElapsedMilliseconds))