Parsing CSV from stream with TextFieldParser always reaches EndOfData - vb.net

During parsing CSV file as a stream from Azure Blob, TextFieldParser always reaches EndOfData immediately, without any data read. The same code, but with the path to same physical file instead of stream works.
Dim storageAccount As CloudStorageAccount = CloudStorageAccount.Parse(AzureStorageConnection)
Dim blobClient As CloudBlobClient = storageAccount.CreateCloudBlobClient()
Dim BlobList As IEnumerable(Of CloudBlockBlob) = blobClient.GetContainerReference("containername").ListBlobs().OfType(Of CloudBlockBlob)
For Each blb In BlobList
Dim myList As New List(Of MyBusinessObject)
Using memoryStream = New MemoryStream()
blb.DownloadToStream(memoryStream)
Using Reader As New FileIO.TextFieldParser(memoryStream)
Reader.TextFieldType = FileIO.FieldType.FixedWidth
Reader.SetFieldWidths(2, 9, 10)
Dim currentRow As String()
While Not Reader.EndOfData
Try
currentRow = Reader.ReadFields()
myList.Add(New GsmXFileRow() With {
' code to read currentRow and add elements to myList
})
Catch ex As FileIO.MalformedLineException
End Try
End While
End Using
End Using
Next
I have also tried to convert MemoryStream to TextReader
Dim myTextReader As TextReader = New StreamReader(memoryStream)
and then passing myTextReader into TextFieldParser, but this does not work either.
Using Reader As New FileIO.TextFieldParser(myTextReader)

I see this:
Value of Length property equals file size
and this:
'Position` property has same value
That means at the start of the loop, the MemoryStream has already advanced to the end of the stream. Just set Position back to 0, and you should be in a better place.
However, there may be another issue here, too. That stream data is binary with some unknown encoding. The TextFieldParser wants to work with Text. You need a way to give the TextFieldParser information about what encoding is used.
In this case, I recommend a StreamReader. This type inherits from TextReader, so you can use it with the TextFieldParser :
Dim storageAccount As CloudStorageAccount = CloudStorageAccount.Parse(AzureStorageConnection)
Dim blobClient As CloudBlobClient = storageAccount.CreateCloudBlobClient()
Dim BlobList As IEnumerable(Of CloudBlockBlob) = blobClient.GetContainerReference("containername").ListBlobs().OfType(Of CloudBlockBlob)
Dim myList As New List(Of MyBusinessObject)
For Each blb In BlobList
'Several constructor overloads allow you to specify the encoding here
Using blobData As New StreamReader(New MemoryStream())
blb.DownloadToStream(blobData.Stream)
'Fix the position problem
blobData.Stream.Position = 0
Using Reader As New FileIO.TextFieldParser(blogData)
Reader.TextFieldType = FileIO.FieldType.FixedWidth
Reader.SetFieldWidths(2, 9, 10)
Dim currentRow As String() = Reader.ReadFields()
While Not Reader.EndOfData
Try
myList.Add(New GsmXFileRow() With {
' code to read currentRow and add elements to myList
})
currentRow = Reader.ReadFields()
Catch ex As FileIO.MalformedLineException
End Try
End While
End Using
End Using
Next

Related

Serialising an object(s) to a file and reading back the list of

in my vb.net app I serialise an object to a file. I do this every now and again.
I close my app
I start my app and I try/want to read the list of objects I have written but it does not work.
This is my model:
<Serializable()>
Public Class test
Public FontColor As Color = Color.Black
Public Contrast As Int32 = 0
Public Text As String = ""
End Class
This is how I write to my file:
Dim formatter As New Runtime.Serialization.Formatters.Binary.BinaryFormatter
Dim stream As New FileStream(Application.StartupPath + "\testme.txt", FileMode.Append, FileAccess.Write, FileShare.None)
Dim TextValues As New test
TextValues.Contrast = 10
TextValues.FontColor = Color.AliceBlue
TextValues.Text = "hello andy"
formatter.Serialize(stream, TextValues)
stream.Close()
NB, yes I know I should be using the keyword 'Using' :)
This is how I read it back in:
Dim formatter As New Runtime.Serialization.Formatters.Binary.BinaryFormatter
Dim stream As New FileStream(Application.StartupPath + "\testme.txt", FileMode.Open, FileAccess.Read, FileShare.Read)
REM THIS WORKS OK BUT FOR ONLY 1 OBJECT ******************************
Dim TextValues = CType(formatter.Deserialize(stream), test)
REM *********************************************************************
REM ***THIS JUST ERRORS *************************************************
Dim TextValues2 = CType(formatter.Deserialize(stream), List(Of test))
REM *********************************************************************
stream.Close()
You can create a List(Of TextValue) and serialize/deserialize that if you want to write and read multiple TextValue objects.

SqlFileStream writing in separate thread not working

Learning and testing using Sql FILESTREAM for a web app. A client uploads a large file form the web page which takes 'X' time and when fully uploaded shows 100% complete. However, very large files also take time for SqlFileStream to write to the file system so I want to spin off a thread to complete that part. The code I've got seems to work fine but no data ends up in the filestream file.
I'm wrapping the initial record creation in it's own transaction scope and using a separate transaction scope in the thread. In the threaded routine I have the appropriate PathName() and TransactionContext but I assume I'm missing something while using a thread.
I've commented out the normal SqlFileStream call which works fine. Can you see anything wrong with what I'm doing here?
Public Function StoreFileStream()
Dim json As New Dictionary(Of String, Object)
Dim parms As New FileStreamThreadParameters
If HttpContext.Current.Request.Files.Count > 0 Then
Dim file As HttpPostedFile = HttpContext.Current.Request.Files(0)
If "contentType" <> String.Empty Then
Dim fs As Stream = file.InputStream
Dim br As New BinaryReader(fs)
Dim noBytes As New Byte()
Try
Dim filePath As String = ""
Dim trxContext As Byte() = {}
Dim baseFileId As Integer
Using trxScope As New TransactionScope
Using dbConn As New SqlConnection(DigsConnStr)
Using dbCmd As New SqlCommand("ADD_FileStreamFile", dbConn)
dbConn.Open()
Using dbRdr As SqlDataReader = dbCmd.ExecuteReader(CommandBehavior.SingleRow)
dbRdr.Read()
If dbRdr.HasRows Then
filePath = dbRdr("Path")
trxContext = dbRdr("TrxContext")
baseFileId = dbRdr("BaseFileID")
End If
dbRdr.Close()
End Using
' Code below writes to file, but trying to offload this to a separate thread so user is not waiting
'Using dest As New SqlFileStream(filePath, trxContext, FileAccess.Write)
' fs.CopyTo(dest, 4096)
' dest.Close()
'End Using
End Using
dbConn.Close()
End Using
trxScope.Complete()
End Using ' transaction commits here, not in line above
parms.baseFileId = baseFileId
parms.fs = New MemoryStream
fs.CopyTo(parms.fs)
Dim fileUpdateThread As New Threading.Thread(Sub()
UpdateFileStreamThreaded(parms)
End Sub)
fileUpdateThread.Start()
json.Add("status", "success")
Catch ex As Exception
Elmah.ErrorSignal.FromCurrentContext().Raise(ex)
json.Add("status", "failure")
json.Add("msg", ex.Message)
json.Add("procedure", System.Reflection.MethodBase.GetCurrentMethod.Name)
End Try
Else
json.Add("status", "failure")
json.Add("msg", "Invalid file type")
json.Add("procedure", System.Reflection.MethodBase.GetCurrentMethod.Name)
End If
End If
Return json
End Function
Public Class FileStreamThreadParameters
Public Property baseFileId As Integer
Public fs As Stream
End Class
Private Sub UpdateFileStreamThreaded(parms As FileStreamThreadParameters)
Dim filePath As String = ""
Dim trxContext As Byte() = {}
Try
Using trxScope As New TransactionScope
Using dbConn As New SqlConnection(DigsConnStr)
Using dbCmd As New SqlCommand("SELECT FileBytes.PathName() 'Path', GET_FILESTREAM_TRANSACTION_CONTEXT() 'TrxContext' FROM FileStreamFile WHERE Id = " & parms.baseFileId, dbConn)
dbConn.Open()
Using dbRdr As SqlDataReader = dbCmd.ExecuteReader(CommandBehavior.SingleRow)
dbRdr.Read()
If dbRdr.HasRows Then
filePath = dbRdr("Path")
trxContext = dbRdr("TrxContext")
End If
dbRdr.Close()
Using dest As New SqlFileStream(filePath, trxContext, FileAccess.Write)
parms.fs.CopyTo(dest, 4096)
dest.Close()
End Using
End Using
End Using
dbConn.Close()
End Using
trxScope.Complete()
End Using
Catch ex As Exception
'Elmah.ErrorSignal.FromCurrentContext().Raise(ex)
End Try
End Sub
Obviously this is a complex issue. I actually got it to work with the code below. However I eventually abandoned using SQL FILESTREAM altogether due to too much complexity in getting it all set up.
This is an existing web application with the sql server on a different box. After I got the filestreaming to work the next hurdle was authentication setup. Filestream requires Integrated Security on your connection string. Even with windows authentication on our Intranet app, I could not get the web app to use the windows credentials when connecting to the database. It always seemed to use the computer name or the app pool service. I tried many many examples I found on the net and here to no avail. Even if I got that to work then I would want to use and Active Directory group over individual logins which looked to be another hurdle.
This app stores documents in a varbinary column so that full text search can be enabled at some point. The issue was with large files which are generally pictures or videos. Since you can't search text on those anyway the strategy now is to store those on the file system and all other searchable docs (.docx, .pptx, etc) will still be stored in the varbinary. I'm actually sad that I could not get filestream to work as it seems like the ideal solution. I'll come back to it some day but it really should not be so frickin complicated. :-(
The code I got working is:
Dim filePath As String = ""
Dim trxContext As Byte() = {}
Dim baseFileId As Integer
Using trxScope As New TransactionScope
Using dbConn As New SqlConnection(DigsFSConnStr)
Using dbCmd As New SqlCommand("NEW_FileStreamBaseFile", dbConn)
dbCmd.CommandType = CommandType.StoredProcedure
dbCmd.Parameters.AddWithValue("#Title", fileDesc)
dbCmd.Parameters.AddWithValue("#Summary", summary)
dbCmd.Parameters.AddWithValue("#Comments", comments)
dbCmd.Parameters.AddWithValue("#FileName", uploadedFileName)
dbCmd.Parameters.AddWithValue("#ContentType", contentType)
dbCmd.Parameters.AddWithValue("#FileExt", ext)
'dbCmd.Parameters.AddWithValue("#FileBytes", noBytes) ' now that were offloading the file byte storage to a thread
dbCmd.Parameters.AddWithValue("#UploadedByResourceID", uploadedByResourceID)
dbCmd.Parameters.AddWithValue("#UploadedByShortName", uploadedByShortName)
dbCmd.Parameters.AddWithValue("#FileAuthor", fileAuthor)
dbCmd.Parameters.AddWithValue("#TagRecID", tagRecID)
dbCmd.Parameters.AddWithValue("#UserID", samAccountName)
dbCmd.Parameters.AddWithValue("#FileDate", fileDate)
dbCmd.Parameters.AddWithValue("#FileType", fileType)
dbCmd.Parameters.AddWithValue("#FileTypeRecID", fileTypeRecId)
' Save to file system too for xod conversion
file.SaveAs(HttpContext.Current.Server.MapPath("~/files/uploaded/") & uploadedFileName)
dbConn.Open()
Using dbRdr As SqlDataReader = dbCmd.ExecuteReader(CommandBehavior.SingleRow)
dbRdr.Read()
If dbRdr.HasRows Then
filePath = dbRdr("Path")
trxContext = dbRdr("TrxContext")
json.Add("baseFileId", dbRdr("BaseFileID"))
virtualFileRecId = dbRdr("VirtualFileRecID")
dbStatus = dbRdr("status")
If dbStatus = "failure" Then
json.Add("msg", dbRdr("msg"))
End If
End If
dbRdr.Close()
End Using
' Prepare and start Task thread to write the file
If dbStatus = "success" Then
bytes = br.ReadBytes(fs.Length)
Dim task As New System.Threading.Tasks.Task(
Sub()
UpdateNewFileStreamBytes(virtualFileRecId, bytes)
End Sub)
task.Start()
json.Add("status", "success")
Else
json.Add("status", "failure")
End If
End Using
dbConn.Close()
End Using
trxScope.Complete()
End Using ' transaction commits here, not in line above
With the task procedure of:
Private Sub UpdateNewFileStreamBytes(virtualFileRecId As Integer, fileBytes As Byte())
Dim filePath As String = ""
Dim trxContext As Byte() = {}
Try
Using trxScope As New TransactionScope
Using dbConn As New SqlConnection(DigsFSConnStr)
Using dbCmd As New SqlCommand("UPD_FileStreamBaseFile", dbConn)
dbCmd.CommandType = CommandType.StoredProcedure
dbCmd.Parameters.AddWithValue("#VirtualFileRecID", virtualFileRecId)
dbConn.Open()
Using dbRdr As SqlDataReader = dbCmd.ExecuteReader(CommandBehavior.SingleRow)
dbRdr.Read()
If dbRdr.HasRows Then
filePath = dbRdr("Path")
trxContext = dbRdr("TrxContext")
End If
dbRdr.Close()
Using dest As New SqlFileStream(filePath, trxContext, FileAccess.Write)
dest.Write(fileBytes, 0, fileBytes.Length)
dest.Close()
End Using
End Using
End Using
dbConn.Close()
End Using
trxScope.Complete()
End Using
Catch ex As Exception
Elmah.ErrorSignal.FromCurrentContext().Raise(ex)
End Try

VB.NET MD5 Comparing using large files

I'm trying to create a program in VB.NET 2010 that has a simple functionality.
It has to compare thousands of MD5 Hashes stored in a text file to current file MD5 Hash that program is automatically calculating while opening the file.
Like a antivirus scanner.
Actually my program use ReadAllText system to add all hashes from text file to a textbox and then it compare them.
Everyting is okay when Md5 database (that text file with hashes) is small but when the file get bigger, my program simply freezes after opening so I decided to use ReadLine instead of ReadAllText.
Now I can't use textbox any longer so give me a way how I can compare them.
I tried a way but it don't seem to work.
Here is my code using ReadLine.
Problem is on If Read contains(buff.tostring) it used to be
If textbox.text contains (buff.toString)
Here is the code
Try
TextBox2.Text = e.FullPath
ListBox3.Items.Add(TextBox2.Text.ToString)
Me.OpenFileDialog1.FileName = ""
Dim reader As StreamReader = New StreamReader("def.txt")
Dim read = reader.ReadLine()
Dim md5 As New MD5CryptoServiceProvider
Dim f As New FileStream(e.FullPath, FileMode.Open, FileAccess.Read, FileShare.Read, &H2000)
f = New FileStream(e.FullPath, FileMode.Open, FileAccess.Read, FileShare.Read, &H2000)
md5.ComputeHash(f)
Dim hash As Byte() = md5.Hash
Dim buff As New StringBuilder
Dim hashByte As Byte
For Each hashByte In hash
buff.Append(String.Format("{0:X2}", hashByte))
Next
f.Close()
If read.Contains(buff.ToString) Then
Me.OpenFileDialog1.FileName = e.FullPath
Form2.ShowDialog()
End If
Catch exception1 As Exception
ProjectData.SetProjectError(exception1)
Dim ex As Exception = exception1
ProjectData.ClearProjectError()
End Try
I would first create functions to split the functionality. It's a lot easier to understand the code afterward.
Store the hashes in a list, this list can then be cached is needed.
Try
TextBox2.Text = e.FullPath
ListBox3.Items.Add(TextBox2.Text.ToString)
Me.OpenFileDialog1.FileName = ""
Dim allHash As List(Of String) = GetAllHash()
Dim curHash As String = GetFileHash(e.FullPath)
If allHash.Contains(curHash) Then
Me.OpenFileDialog1.FileName = e.FullPath
Form2.ShowDialog()
End If
Catch exception1 As Exception
ProjectData.SetProjectError(exception1)
Dim ex As Exception = exception1
ProjectData.ClearProjectError()
End Try
Function GetAllHash() As List(Of String)
' Store the data in a list instead
Return System.IO.File.ReadAllLines("def.txt").ToList()
End Function
Function GetFileHash(ByVal filename As String) As String
Dim md5 As New MD5CryptoServiceProvider
' Only open the file once
Dim f As New FileStream(filename, FileMode.Open, FileAccess.Read, FileShare.Read, &H2000)
md5.ComputeHash(f)
Dim hash As Byte() = md5.Hash
Dim buff As New StringBuilder
For Each hashByte As Byte In hash
buff.Append(String.Format("{0:X2}", hashByte))
Next
f.Close()
Return buff.ToString()
End Function
I didn't compile this code, this is just a example to show you what can be done.

How do I reload the formatted text into RTB?

I have a RTB in VB.NET, and put an event handler to save the formatted text into a file after encrypting it. However, I can't figure out how to reload the formatting - when I open it, it shows the symbols of formatting instead of formatted text. Here's my code:
Dim FileName As String = TextBox1.Text
File.Delete(FileName)
Dim EncryptElement As New TripleDESCryptoServiceProvider
EncryptElement.Key = {AscW("B"c), AscW("A"c), AscW("1"c), AscW("R"c), AscW("3"c), AscW("9"c), AscW("G"c), AscW("V"c), AscW("5"c), AscW("S"c), AscW("P"c), AscW("0"c), AscW("L"c), AscW("Z"c), AscW("4"c), AscW("M"c)} '128 bit Key
EncryptElement.IV = {AscW("N"c), AscW("B"c), AscW("5"c), AscW("3"c), AscW("G"c), AscW("L"c), AscW("2"c), AscW("Q"c)} ' 64 bit Initialization Vector
Dim fStream As FileStream = File.Open(FileName, FileMode.OpenOrCreate)
Dim cStream As New CryptoStream(fStream, New TripleDESCryptoServiceProvider().CreateEncryptor(EncryptElement.Key, EncryptElement.IV), CryptoStreamMode.Write)
Dim sWriter As New StreamWriter(cStream)
sWriter.WriteLine(RichTextBox1.Rtf)
sWriter.Close()
cStream.Close()
fStream.Close()
The above code is for saving, and the below code is for opening.
Dim FileName As String = TextBox1.Text
Dim DecryptElement As New TripleDESCryptoServiceProvider
DecryptElement.Key = {AscW("B"c), AscW("A"c), AscW("1"c), AscW("R"c), AscW("3"c), AscW("9"c), AscW("G"c), AscW("V"c), AscW("5"c), AscW("S"c), AscW("P"c), AscW("0"c), AscW("L"c), AscW("Z"c), AscW("4"c), AscW("M"c)}
DecryptElement.IV = {AscW("N"c), AscW("B"c), AscW("5"c), AscW("3"c), AscW("G"c), AscW("L"c), AscW("2"c), AscW("Q"c)}
Dim fStream As FileStream = File.Open(FileName, FileMode.OpenOrCreate)
Dim cStream As New CryptoStream(fStream, New TripleDESCryptoServiceProvider().CreateDecryptor(DecryptElement.Key, DecryptElement.IV), CryptoStreamMode.Read)
Dim sReader As New StreamReader(cStream)
Dim DecryptedData As String = ""
DecryptedData = sReader.ReadToEnd
RichTextBox1.AppendText(DecryptedData)
RichTextBox1.Enabled = True
Button1.Text = "OK"
sReader.Close()
cStream.Close()
fStream.Close()
Where is the problem?
You need RichTextBox1.SaveFile(SomeStream, RichTextBoxStreamType) I think StreamWriter is stuffing it up.
Oh and seeing as you just gave the world the keys for your encryption you might want to come up with a new one...
Added after comment not proven though.
Replace these two lines in your save routine
Dim sWriter As New StreamWriter(cStream)
sWriter.WriteLine(RichTextBox1.Rtf)
with
RichTextBox1.SaveFile(cStream,RichTextBoxStreamType.RichText);
and get rid of swriter.Close()
I think.

trouble returning datatable within a try statement

I have the following piece of code that I am using to try and rip a csv file and turn it into a datatable. My problem is that the debugger never makes it to the return statement. Everything is appending to the datatable correctly, so I know that part works. Any idea's on what I can do to trouble shoot this further. Also, if you know of a simpler way to turn a import a csv file to a datatable I'd be very interested in learning about it.
Thanks!
Public Function loadCSVTableII() As DataTable
Dim dt As New DataTable("TableII")
Dim line As String = String.Empty
Dim counter As Integer = 0
Dim reader As New StreamReader(pathTableTwo)
Try
While Not IsNothing(line)
line = reader.ReadLine()
Dim lineSep As String() = line.Split(New Char() {","c})
If Not counter = 0 Then
dt.Rows.Add(lineSep)
counter += 1
Else
For Each value As String In lineSep
dt.Columns.Add(value)
Next
counter += 1
End If
End While
'cursor never gets to this code block...
Dim primarykey(0) As DataColumn
primarykey(0) = dt.Columns("Ages")
dt.PrimaryKey = primarykey
Return dt
Catch ex As Exception
Throw
End Try
End Function
Update: It is erroring out on this line in the code.
Dim lineSep As String() = line.Split(New Char() {","c})
It say that the Object reference is not set to an instance of an object. What's weird though is that it works through the whole data table fine. Could it be that the while loop is not terminating at the end of the file?
Try changing your While loop to handle the end of stream condition. It's not very clear what the IsNothing function is doing in your code.
While Not reader.EndOfStream
line = reader.ReadLine
'// Dim lineSep As String() = line.Split(New Char() {","c})
For your line split, in VB.Net, it's simple to just do this:
Dim lineSep As String() = line.Split(",")
You can use OLEDB provider for this.
string query = "SELECT Symbol, [Name of Company], FROM [just file name with extension]";
string connStr = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + [csv file path without file name] + ";" + "Extended Properties=’text;HDR=YES;’";
//create dataadapter object
OleDbDataAdapter adapter = new OleDbDataAdapter(query, connStr);
// create table
DataTable dtSymbolDetails = new DataTable("ScriptDetails");
dtSymbolDetails.Columns.Add("Symbol");
dtSymbolDetails.Columns.Add("Name of Company");
// fill the table with data using adapter
adapter.Fill(dtDetails);