I am trying to figure out how to read a text file into an array in vb.net - vb.net

Question

I'm trying to create a program in vb.net (forms) to process data from a UVvis spectrometer.
The txt file output looks as following.
"180809_QuartzRefTrans.spc - RawData"
"Wavelength nm.","T%"
400.00,90.822
401.00,90.800
402.00,90.823
403.00,90.811
404.00,90.803
405.00,90.804
406.00,90.816
407.00,90.811
408.00,90.833
409.00,90.837
410.00,90.847
411.00,90.827
412.00,90.839
413.00,90.851
414.00,90.828
415.00,90.879
416.00,90.846
and so on.
What I am trying to do is read the data it into an array so I can manipulate the columns. I need to be able to skip the first two lines so that all I have is numerical data. I also need it to sort the array from lowest to highest (wavelengths). Sometimes we run from 800->200 nm then accidentally put in 200->800 nm
Imports System.IO
Imports System.Windows.Forms.DataVisualization.Charting
Public Class Form1
Public Class RefTrans
Public Property Wavelength As Double
Public Property Transpercent As Double
End Class
Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
Dim strtext As String
OpenFileDialog1.Title = "Open Text Files"
OpenFileDialog1.ShowDialog()
strtext = OpenFileDialog1.FileName
TextBox1.Text = My.Computer.FileSystem.GetName(strtext)
Label1.Text = My.Computer.FileSystem.GetName(strtext)
Dim line1 As String
Dim output1 As New ArrayList
Using sr As New IO.StreamReader(strtext)
sr.ReadLine()
sr.ReadLine()
Do While sr.Peek() >= 0
line1 = sr.ReadLine()
output1.Add(line1)
Loop
End Using
If strtext <> "" Then
Dim SR As New StreamReader(strtext)
SR.ReadLine()
Do Until SR.EndOfStream
TextBox3.Text = TextBox3.Text & SR.ReadLine & vbCrLf
Loop
SR.Close()
End If
Dim data1 = IO.File.ReadLines(strtext).
Skip(2).
Select(Function(line)
Dim parts = line.Split(","c)
Return New RefTrans With {.Wavelength = CDbl(parts(0)),
.Transpercent = CDbl(parts(1))}
End Function).
ToArray() = line.Split(","c)
End Sub
End Class

Answer 1

Here's an example that will get you an array of Tuple(Of Double, Double):
Dim data = IO.File.ReadLines(filePath).
Skip(2).
Select(Function(line)
Dim parts = line.Split(","c)
Return Tuple.Create(CDbl(parts(0)), CDbl(parts(1)))
End Function).
ToArray()
The IO.File.ReadLines method will read the lines of the file. Where the ReadAllLines method reads the entire file and then returns the lines as a String array, the ReadLines method exposes the lines of the file as an enumerable list and the lines only get read from the file as you use them. Particularly for big files, ReadLines is more efficient than ReadAllLines because it does create an intermediate array first that you don't actually want.
The Skip method allows you to access the items in a list starting at a particular index. In the case of reading a file, you need to read every line regardless but call Skip(2) on the result of File.ReadLines means that the first two lines of the file will be discarded after reading and any subsequent processing will only but done on lines from the third.
The Select method is basically a transformation. It says create an output list containing an item for every item in the input list where the output item is the result of a transformation of the input item. The transformation is defined by the function you provide. In this case, a line from the file is the input and the transformation is to split that line on the comma, convert the two substrings to Double values and then create a Tuple containing those two Double values.
The ToArray method takes any enumerable list, i.e. any object that implements IEnumerable(Of T), and returns an array containing the items from that list. In this case, Select returns an IEnumerable(Of Tuple(Of Double, Double)) so ToArray returns a Tuple(Of Double, Double) array.
If you wanted to write that all out long-hand then it would look like this:
'Get all the lines of the file.
Dim step1 As IEnumerable(Of String) = IO.File.ReadLines(filePath)
'Skip the first two lines.
Dim step2 As IEnumerable(Of String) = step1.Skip(2)
'Split each line into two substrings.
Dim step3 As IEnumerable(Of String()) = step2.Select(Function(line) line.Split(","c))
'Convert substrings to numbers and combine.
Dim step4 As IEnumerable(Of Tuple(Of Double, Double)) = step3.Select(Function(parts) Tuple.Create(CDbl(parts(0)), CDbl(parts(1))))
'Create an array of Tuples.
Dim data As Tuple(Of Double, Double)() = step4.ToArray()
A Tuple is basically a general-purpose object for grouping values together. You can use a Tuple for one-off cases rather than defining your own class. You may prefer to define your own class in all cases though, and you should define your own class where you will use the data extensively. In this case, your own class might look like this:
Public Class RefTrans
Public Property Wavelength As Double
Public Property TransPercent As Double
End Class
and the code would then become:
Dim data = IO.File.ReadLines(filePath).
Skip(2).
Select(Function(line)
Dim parts = line.Split(","c)
Return New RefTrans With {.Wavelength = CDbl(parts(0)),
.TransPercent = CDbl(parts(1))}
End Function).
ToArray()
In that case, you have properties Wavelength and TransPercent on each element of your array. If you use Tuples then the properties have the generic names Item1 and Item2.
Once you have the array, you can use any appropriate overload of Array.Sort to do the sorting, e.g.
Array.Sort(data, Function(d1, d2) d1.Item1.CompareTo(d2.Item1))
That will sort the Tuples by comparing their Item1 properties, which contain the wavelength values. If you're using your own class then obviously you specify your own property, e.g. the Wavelength property in the RefTrans class I showed.