Finding And Removing Duplicates from text file using vb.net - vb.net

Question

I have a huge text file in which a large number of duplication are occurred. The duplications are as follows.
Total Posts 16
Pin Code = GFDHG
TITLE = Shop Signs/Projection Signs/Industrial Signage/Restaurant signs/Menu Boards&Box in London
DATE = 12-09-2012
Tracking Key # 85265E712050-15207427406854753
Total Posts 16
Pin Code = GFDHG
TITLE = Shop Signs/Projection Signs/Industrial Signage/Restaurant signs/Menu Boards&Box in London
DATE = 12-09-2012
Tracking Key # 85265E712050-15207427406854753
Total Posts 2894
Pin Code = GFDHG
TITLE = Shop Signs/Projection Signs/Industrial Signage/Restaurant signs/Menu Boards&Box in London
DATE = 15-09-2012
Tracking Key # 85265E712050-152797637654753
Total Posts 2894
Pin Code = GFDHG
TITLE = Shop Signs/Projection Signs/Industrial Signage/Restaurant signs/Menu Boards&Box in London
DATE = 15-09-2012
Tracking Key # 85265E712050-152797637654753
and so on upto 4000 total posts are there in this text file. I want that my program match the total post 6 to all total post that occurred in file and where find the duplicate , then programatically remove that duplicate and also delete the next 7 lines of that duplicate. Thank you

Answer 1

Assuming the formatting is consistent (i.e. each logged event in the file uses 6 total lines of text), then if you're looking to remove duplicates from the file you just need to do something like this:
Sub DupClean(ByVal fpath As String) 'fpath is the FULL file path, i.e. C:\Users\username\Documents\filename.txt
Dim OrigText As String = ""
Dim CleanText As String = ""
Dim CText As String = ""
Dim SReader As New System.IO.StreamReader(fpath, System.Text.Encoding.UTF8)
Dim TxtLines As New List(Of String)
Dim i As Long = 0
Dim writer As New System.IO.StreamWriter(Left(fpath, fpath.Length - 4) & "_clean.txt", False) 'to overwrite the text inside the same file simply use StreamWriter(fpath)
Try
'Read in the text
OrigText = SReader.ReadToEnd
'Parse the text at new lines to allow selecting groups of 6 lines
TxtLines.AddRange(Split(OrigText, Chr(10))) 'may need to change the Chr # to look for depending on if 10 or 13 is used when the file is generated
Catch ex As Exception
MsgBox("Encountered an error while reading in the text file contents and parsing them. Details: " & ex.Message, vbOKOnly, "Read Error")
End
End Try
Try
'Now we iterate through blocks of 6 lines
Do While i < TxtLines.Count
'Set CText to the next 6 lines of text
CText = TxtLines.Item(i) & Chr(10) & TxtLines.Item(i + 1) & Chr(10) & TxtLines.Item(i + 2) & Chr(10) & TxtLines.Item(i + 3) & Chr(10) & TxtLines.Item(i + 4) & Chr(10) & TxtLines.Item(i + 5)
'Check if CText is already present in CleanText
If Not (CleanText.Contains(CText)) Then
'Add CText to CleanText
If CleanText.Length = 0 Then
CleanText = CText
Else
CleanText = CleanText & Chr(10) & CText
End If
End If 'else the text is already present and we don't need to do anything
i = i + 6
Loop
Catch ex As Exception
MsgBox("Encountered an error while running cleaning duplicates from the read in text. The application was on the " & i & "-th line of text when the following error was thrown: " & ex.Message, _
vbOKOnly, "Comparison Error")
End
End Try
Try
'Write out the clean text
writer.Write(CleanText)
Catch ex As Exception
MsgBox("Encountered an error writing the cleaned text. Details: " & ex.Message & Chr(10) & Chr(10) & "The cleaned text was " & CleanText, vbOKOnly, "Write Error")
End Try
End Sub
If the format isn't consistent you'll need to get fancier and define rules to tell which lines to add to CText on any given pass through the loop, but without context I wouldn't be able to give you any ideas as to what those may be.