This is a very simple question.
I am trying to add items to the Inventory class list from a .txt file. However i can't use the word 'New' in this line otherwise it give a syntax error:
New Inventory("Rod", 1)
Function GetInventory() As IEnumerable(Of Inventory)
If System.IO.File.Exists(loc) = True Then
If System.IO.File.ReadAllText(loc).Count > 0 Then
Dim file As System.IO.StreamReader
file = My.Computer.FileSystem.OpenTextFileReader(loc)
Do While file.Peek() >= 0
New Inventory("Rod", 1)
Loop
Return New Inventory
file.Close()
End If
End If
End Function
How do I go about this???
Thank you.
With what you have, you're reading the entire file into memory twice. That's crazy wasteful.
Also, you don't need the .Exists() check here. The file system is volatile, meaning it's possible for the file to cease to be available between when you check .Exists() and when you try to access the file. A good program will still have a good exception handler for when the file does not exist, and now that you have an exception handler, the .Exists() check is just redundant. It's not saving you the performance hit you likely think it is. Also, this method is probably the wrong place to handle the exception. That's usually better to do in the calling code somewhere, meaning you can skip error checking here completely.
You can get this whole method down to a single statement that will cut your execution time in half:
Function GetInventory(ByVal loc As String) As IEnumerable(Of Inventory)
Return IO.File.ReadLines(loc).Select(Function(i) New Inventory(i, 1))
End Function
Related
It is a very basic code that I'm using for my college coursework. Unfortunately, we're only allowed to use StreamRead and StreamWrite type functions for marking reasons. I'm unsure as to how to basically rewrite the line already in the file with a new value. I know I could delete the file and recreate but we also get marked down for that sort of thing. Any ideas?
Private Function WriteTaxableIncome()
Dim TaxableIncomeStreamWriter As IO.StreamWriter
TaxableIncomeStreamWriter = New IO.StreamWriter("C:\Computing\Coursework\TaxableIncome.txt", True)
TaxableIncomeStreamWriter.WriteLine(TaxableIncome)
TaxableIncomeStreamWriter.Close()
End Function
When you create the StreamWriter instance, you have called the constructor with New (path As String, append As Boolean), and passed True into the append argument, so that will make the writer append to the file, instead of overwrite it. Seems like the exact opposite of what you want to do. Instead, pass False. Also, use a Using block which will Open and Close the writer. It automatically calls Dispose so you can be sure no part of it is left open.
Using writer As New IO.StreamWriter("C:\Computing\Coursework\TaxableIncome.txt", False)
writer.WriteLine(TaxableIncome)
End Using
I have a text file that is 125Mb in size, it contains 2.2 million records. I have another text file which doesn't match the original but I need to find out where it differs. Normally, with a smaller file I would read each line and process it in some way, or read the whole file into a string and do likewise, however the two files are too big for that and so I would like to create something to achieve my goal. Here's what I currently have.. excuse the mess of it.
Private Sub refUpdateBtn_Click(sender As Object, e As EventArgs) Handles refUpdateBtn.Click
Dim refOrig As String = refOriginalText.Text 'Original Reference File
Dim refLatest As String = refLatestText.Text 'Latest Reference
Dim srOriginal As StreamReader = New StreamReader(refOrig) 'start stream of original file
Dim srLatest As StreamReader = New StreamReader(refLatest) 'start stream of latest file
Dim recOrig, recLatest, baseDIR, parentDIR, recOutFile As String
baseDIR = vb.Left(refOrig, InStrRev(refOrig, ".ref") - 1) 'find parent folder
parentDIR = Path.GetDirectoryName(baseDIR) & "\"
recOutFile = parentDIR & "Updated.ref"
Me.Text = "Processing Reference File..." 'update the application
Update()
If Not File.Exists(recOutFile) Then
FileOpen(55, recOutFile, OpenMode.Append)
FileClose(55)
End If
Dim x As Integer = 0
Do While srLatest.Peek() > -1
Application.DoEvents()
recLatest = srLatest.ReadLine
recOrig = srOriginal.ReadLine ' check the original reference file
Do
If Not recLatest.Equals(recOrig) Then
recOrig = srOriginal.ReadLine
Else
FileOpen(55, recOutFile, OpenMode.Append)
Print(55, recLatest & Environment.NewLine)
FileClose(55)
x += 1
count.Text = "Record No: " & x
count.Refresh()
srOriginal.BaseStream.Seek(0, SeekOrigin.Begin)
GoTo 1
End If
Loop
1:
Loop
srLatest.Close()
srOriginal.Close()
FileClose(55)
End Sub
It's got poor programming and scary loops, but that's because I'm not a professional coder, just a guy trying to make his life easier.
Currently, this uses a form to insert the original file and the latest file and outputs each line that matches into a new file. This is less than perfect, but I don't know how to cope with the large file sizes as streamreader.readtoend crashes the program. I also don't need the output to be a copy of the latest input, but I don't know how to only output the records it doesn't find. Here's a sample of the records each file has:
doc:ARCHIVE.346CCBD3B06711E0B40E00163505A2EF
doc:ARCHIVE.346CE683B29811E0A06200163505A2EF
doc:ARCHIVE.346CEB15A91711E09E8900163505A2EF
doc:ARCHIVE.346CEC6AAA6411E0BEBB00163505A2EF
The program I have currently works... to a fashion, however I know there are better ways of doing it and I'm sure much better ways of using the CPU and memory, but I don't know this level of programming. All I would like is for you to take a look and offer your best answers to all or some of the code. Tell me what you think will make it better, what will help with one line, or all of it. I have no time limit on this because the code works, albeit slowly, I would just like someone to tell me where my code could be better and what I could do to get round the huge file sizes.
Your code is slow because it is doing a lot of file IO. You're on the right track by reading one line at a time, but this can be improved.
Firstly, I've created some test files based off the data that you provided. Those files contain three million lines and are about 130 MB in size (2.2 million records was less than 100 MB so I've increased the number of lines to get to the file size that you state).
Reading the entire file into a single string uses up about 600 MB of memory. Do this with two files (which I assume you were doing) and you have over 1GB of memory used, which may have been causing the crash (you don't say what error was shown, if any, when the crash occurred, so I can only assume that it was an OutOfMemoryException).
Here's a few tips before I go through your code:
Use Using Blocks
This won't help with performance, but it does make your code cleaner and easier to read.
Whenever you're dealing with a file (or anything that implements the IDisposable interface), it's always a good idea to use a Using statement. This will automatically dispose of the file (which closes the file), even if an error happens.
Don't use FileOpen
The FileOpen method is outdated (and even stated as being slow in its documentation). There are better alternatives that you are already (almost) using: StreamWriter (the cousin of StreamReader).
Opening and closing a file two million times (like you are doing inside your loop) won't be fast. This can be improved by opening the file once outside the loop.
DoEvents() is evil!
DoEvents is a legacy method from back in the VB6 days, and it's something that you really want to avoid, especially when you're calling it two million times in a loop!
The alternative is to perform all of your file processing on a separate thread so that your UI is still responsive.
Using a separate thread here is probably overkill, and there are a number of intricacies that you need to be aware of, so I have not used a separate thread in the code below.
So let's look at each part of your code and see what we can improve.
Creating the output file
You're almost right here, but you're doing some things that you don't need to do. GetDirectoryName works with file names, so there's no need to remove the extension from the original file name first. You can also use the Path.Combine method to combine a directory and file name.
recOutFile = Path.Combine(Path.GetDirectoryName(refOrig), "Updated.ref")
Reading the files
Since you're looping through each line in the "latest" file and finding a match in the "original" file, you can continue to read one line at a time from the "latest" file.
But instead of reading a line at a time from the "original" file, then seeking back to the start when you find a match, you will be better off reading all of those lines into memory.
Now, instead of reading the entire file into memory (which took up 600 MB as I mentioned earlier), you can read each line of the file into an array. This will use up less memory, and is quite easy to do thanks to the File class.
originalLines = File.ReadAllLines(refOrig)
This reads all of the lines from the file and returns a String array. Searching through this array for matches will be slow, so instead of reading into an array, we can read into a HashSet(Of String). This will use up a bit more memory, but it will be much faster to seach through.
originalLines = New HashSet(Of String)(File.ReadAllLines(refOrig))
Searching for matches
Since we now have all of the lines from the "original" line in an array or HashSet, searching for a line is very easy.
originalLines.Contains(recLatest)
Putting it all together
So let's put all of this together:
Private Sub refUpdateBtn_Click(sender As Object, e As EventArgs)
Dim refOrig As String
Dim refLatest As String
Dim recOutFile As String
Dim originalLines As HashSet(Of String)
refOrig = refOriginalText.Text 'Original Reference File
refLatest = refLatestText.Text 'Latest Reference
recOutFile = Path.Combine(Path.GetDirectoryName(refOrig), "Updated.ref")
Me.Text = "Processing Reference File..." 'update the application
Update()
originalLines = New HashSet(Of String)(File.ReadAllLines(refOrig))
Using latest As New StreamReader(refLatest),
updated As New StreamWriter(recOutFile, True)
Do
Dim line As String
line = latest.ReadLine()
' ReadLine returns Nothing when it reaches the end of the file.
If line Is Nothing Then
Exit Do
End If
If originalLines.Contains(line) Then
updated.WriteLine(line)
End If
Loop
End Using
End Sub
This uses around 400 MB of memory and takes about 4 seconds to run.
I am using the following code to delete all files from a particular folder:
Sub DeleteFiles(Folder As String)
If Directory.Exists(Folder) Then
For Each _file As String In Directory.GetFiles(Folder)
File.Delete(_file)
Next
For Each _folder As String In Directory.GetDirectories(Folder)
DeleteFiles(_folder)
Next
End If
End Sub
Whenever I use the above code to delete all files from "C:\Temp" by calling it using DeleteFiles("C:\Temp"), It deletes all the files successfully, but whenever I try to use the same code for deleting files in "C:\Windows\TEMP\", it breaks the operation saying that the file is in use. I want that the code should not raise an exception and stop deleting the files right-away. If the file cannot be deleted, the code should move on to the next file and try deleting it. This way, it should be able to delete maximum possible files from that directory.
You need to handle exceptions raised by File.Delete() using a Try....Catch... statement. I'm not a VB coder (I'm surprised to find myself answering this question), but something like this should work:
Sub DeleteFiles(Folder As String)
If Directory.Exists(Folder) Then
For Each _file As String In Directory.GetFiles(Folder)
Try
File.Delete(_file)
Catch e As System.IO.IOException
Console.WriteLine(e.Message)
End Try
Next
For Each _folder As String In Directory.GetDirectories(Folder)
DeleteFiles(_folder)
Next
End If
End Sub
This will catch an System.IO.IOException exception, log that it was received, and then ignore it. Note that this will catch a number of other File.Delete() related exceptions such as System.IO.DirectoryNotFoundException, System.IO.PathTooLongException, etc. If you want to catch these, you must add a Catch clause for each before the more general System.IO.IOException. The possible exceptions are listed here, along with an example of using File.Delete() - you just need to read the docs.
You might like to also look at Directory.Delete to recursively delete a directory, its files, and its subdirectories.
You can't stop an exception being thrown. What you need to do is catch the exception, process it as appropriate (which may mean just ignoring it) and move on. That means putting a Try...Catch block inside your first loop. That way, when File.Delete throws an exception, you can catch it, ignore it and the loop will continue.
Be sure to catch only the type of exception that you expect to be thrown though. Otherwise, something completely unexpected may be causing an issue and you're just ignoring it, which is bad. Only ignore exceptions that you reasonably expect and know can be safely ignored.
Dim temp As String = Environment.GetEnvironmentVariable("TEMP")
Dim k As String() = System.IO.Directory.GetFiles(temp)
Dim i As Integer
For i = 0 To k.Length
On Error Resume Next
Kill(k(i))
System.IO.File.Delete(k(i))
Next
I'm not an IT professional so apologies if I've missed something obvious.
When writing a program I add a class SettingsIni that reads a text file of keys and values. I find this method really flexible as settings can be added or changed without altering any code, regardless of what application I have attached it to.
Here's the main code.
Public Shared Sub Load()
Using settingsReader As StreamReader = New StreamReader(System.AppDomain.CurrentDomain.BaseDirectory & "settings.ini")
Do While settingsReader.Peek > -1
Dim line As String = settingsReader.ReadLine
Dim keysAndValues() As String = line.Split("="c)
settingsTable.Add(keysAndValues(0).Trim, keysAndValues(1).Trim)
Loop
End Using
End Sub
Public Shared Function GetValue(ByVal key As String)
Dim value As String = settingsTable(key)
Return value
End Function
This allows you to use a setting within your code by calling the SettingsIni.GetValue method.
For example:
watcher = New FileSystemWatcher(SettingsIni.GetValue("inputDir"), "*" & SettingsIni.GetValue("extn")).
I find this makes my code esay to read.
My problem is the values in this case, inputDir and extn, are typed freehand and not checked by intellisense. I'm always worried that I may make a typo in an infrequently used branch of an application and miss it during testing.
Is there a best practice method for retrieving settings? or a way around these unchecked freehand typed values?
A best practice for your code example would be to use Constants for the possible settings.
Class Settings
Const inputDir as String = "inputDir"
Const extn as String = "extn"
End Class
watcher = New FileSystemWatcher(SettingsIni.GetValue(Settings.inputDir), "*" & SettingsIni.GetValue(Settings.extn))
I assume you are using VB.NET?
If so, there is the handy "Settings"-menu under "my project". It offers a way to store the settings for your program and retrieve them via "my.settings.YOURKEY". The advantage is, that type securtiy is enforced on this level.
Additionally, you can also store "resources" almost the same way - but resources are better suited for strings / pictures etc. But they are expecially good if you want to translate your program.
As for your current problem:
Store the path in the settings, this way you do not need to change alll your code immidiately but you can use your system and never misspell anything.
If it's a number you could do these 3 things:
Check if is numeric - using IsNumeric function
Check if it is whole number - using Int function, like: if Int(number)=number
Check for the valid range, like: if number>=lowerbound and number<=upperbound
It totally depends on you. You are the one to check almost all the things inside quotes, not the intellisense.
But you still use Try-Catch block:
Try
Dim value As String = settingsTable(key)
Return value
Catch ex As Exception
MsgBox(ex.ToString)
Return ""
End Try
So you will get an message box if you are trying to access a non-existing setting that you may have mistyped.
I know, I know, I could have used a for loop, dont tell me anything about that. Please, help!
Private Function LoadSaved() ''//Loads saved clippings if the user wants us to
Dim ZomgSavedClips As StringCollection
If IsDBNull(My.Settings.SavedClips) = False Then ''//If it is null this would return a rather ugly error. Dont want that do we?
ZomgSavedClips = My.Settings.SavedClips ''//ZomgSavedClips name was a joke, I just felt like it.
ZomgSavedClips.Add(" ") ''//This line ought to fix the error, but doesnt
i = 0
While i < ZomgSavedClips.Count ''//This is where the error occurs
ClipListings.Rows.Add(ZomgSavedClips(i))
i = i + 1 ''//First time I wrote this function I forgot this line. Crashed mah comp. Fail.
End While
End If
End Function
The line While i < ZomgSavedClips.Count is bugging, I know that the .count should return null but I even added a blank piece of text just to stop that. Whats up with this? Should I add actual text?
SavedClips is null no? If it is null it could pass the test IsDBNull beacuse the both are not the same
Obviously, My.Settings.SavedClips is still set to Nothing.
SavedClips is regular 'ole null (nothing in VB). Include a check for "My.Settings.SavedClips is nothing". If that evaluates to true then just leave the function.
I even added a blank piece of text just to stop that.
All you did was move where the error happens. You can't call .Add() on a null/Nothing object.
'''<summary>Loads saved clippings if the user wants us to</summary>'
Private Sub LoadSaved() ''//Loads saved clippings if the user wants us to
''//Load saved clips into memory
Dim ZomgSavedClips As StringCollection = My.Settings.SavedClips
If ZomgSavedClips Is Nothing Then ZomgSavedClips = New StringCollection()
''//Apply loaded clips to visible listings
Dim i As Integer
While i < ZomgSavedClips.Count ''
ClipListings.Rows.Add(ZomgSavedClips(i))
i += 1
End While
End Sub
Some notes on this code:
Don't use Function when you mean Sub
Since you'll be selling this code to others, you want to use xml comments at the top so that Visual Studio can give better intellisense helps.
IsDBNull() doesn't do what you think it does.
Yes, you should use a for loop, but since you already commented on that I left the while loop alone with the assumption that there's more code you didn't show us.