Okay, so I'm working in VB.NET, manually writing error logs to log files (yes, I know, I didn't make the call). Now, if the files are over an arbitrary size, when the function goes to write out the new error data, it should start a new file with a new file name.
Here's the function:
Dim listener As New Logging.FileLogTraceListener
listener.CustomLocation = System.Configuration.ConfigurationManager.AppSettings("LogDir")
Dim loc As String = DateTime.UtcNow.Year.ToString + DateTime.UtcNow.Month.ToString + DateTime.UtcNow.Day.ToString + DateTime.UtcNow.Hour.ToString + DateTime.UtcNow.Minute.ToString
listener.BaseFileName = loc
Dim logFolder As String
Dim source As String
logFolder = ConfigurationManager.AppSettings("LogDir")
If ex.Data.Item("Source") Is Nothing Then
source = ex.Source
Else
source = ex.Data.Item("Source").ToString
End If
Dim errorFileInfo As New FileInfo(listener.FullLogFileName)
Dim errorLengthInBytes As Long = errorFileInfo.Length
If (errorLengthInBytes > CType(System.Configuration.ConfigurationManager.AppSettings("maxFileSizeInBytes"), Long)) Then
listener.BaseFileName = listener.BaseFileName + "1"
End If
Dim msg As New System.Text.StringBuilder
If String.IsNullOrEmpty(logFolder) Then logFolder = ConfigurationManager.AppSettings("LogDir")
msg.Append(vbCrLf & "Exception" & vbCrLf)
msg.Append(vbTab & String.Concat("App: AppMonitor | Time: ", Date.Now.ToString) & vbCrLf)
msg.Append(vbTab & String.Concat("Source: ", source, " | Message: ", ex.Message) & vbCrLf)
msg.Append(vbTab & "Stack: " & ex.StackTrace & vbCrLf)
listener.Write(msg.ToString())
listener.Flush()
listener.Close()
I have this executing in a loop for testing purposes, so I can see what happens when it gets (say) 10000 errors in all at once. Again, I know there are better ways to handle this systemically, but this was the code I was told to implement.
How can I reliably get the size of the log file before writing to it, as I try to do above?
Well, as with many things, the answer to this turned out to be "did you read your own code closely" with a side order of "eat something, you need to fix your blood sugar."
On review, I saw that I was always checking BaseFileName and, if it was over the arbitrary limit, appending a character and writing to that file. What I didn't do was check to see if that file or, indeed, other more recent files existed. I've solved the issue be grabbing a directory list of all the files matching the "BaseFileName*" argument in Directory.GetFiles and selecting the most recently accessed one. That ensures that the logger will always select the more current file to write to or -if necessary- use as the base-name for another appended character.
Here's that code:
Dim directoryFiles() As String = Directory.GetFiles(listener.Location.ToString(), listener.BaseFileName + "*")
Dim targetFile As String = directoryFiles(0)
For j As Integer = 1 To directoryFiles.Count - 1 Step 1
Dim targetFileInfo As New FileInfo(targetFile)
Dim compareInfo As New FileInfo(directoryFiles(j))
If (targetFileInfo.LastAccessTimeUtc < compareInfo.LastAccessTimeUtc) Then
targetFile = directoryFiles(j)
End If
Next
Dim errorFileInfo As New FileInfo(listener.Location.ToString() + targetFile)
Dim errorLengthInBytes As Long = errorFileInfo.Length
Related
I coded a simple program that reads from a Textfile Line by Line and If the current readed Line has alphabetics (a-z A-Z) it will write that Line into an other txt file.
If the current readed line doesn't have alphabetics it wont write that line into a new text file.
I created this for the purpose that I have members registering at my website and some of them are using only numbers as Username. I will filter them out and only save the alphabetic Names. (Focus on this Project please I know i could just use php stuff)
That works great already but it takes a while to read line by line and write into the other text file (Write speed 150kb in 1 Minute - Its not my drive I have a fast ssd).
So I wonder if there is a faster way. I could "readalllines" first but on large files it just freezes my program so I don't know if that works too (I want to focus on large +1gb files)
This is my code so far:
If System.IO.File.Exists(FILE_NAME) = True Then
Dim objReader As New System.IO.StreamReader(FILE_NAME)
Do While objReader.Peek() <> -1
Dim myFile As New FileInfo(output)
Dim sizeInBytes As Long = myFile.Length
If sizeInBytes > splitvalue Then
outcount += 1
output = outputold + outcount.ToString + ".txt"
File.Create(output).Dispose()
End If
count += 1
TextLine = objReader.ReadLine() & vbNewLine
Console.WriteLine(TextLine)
If CheckForAlphaCharacters(TextLine) Then
File.AppendAllText(output, TextLine)
Else
found += 1
Label2.Text = "Removed: " + found.ToString
TextBox1.Text = TextLine
End If
Label1.Text = "Checked: " + count.ToString
Loop
MessageBox.Show("Finish!")
End If
First of all, as hinted by #Sean Skelly updating UI controls - repeatedly - is an expensive operation.
But your bigger problem is File.AppendAllText:
If CheckForAlphaCharacters(TextLine) Then
File.AppendAllText(output, TextLine)
Else
found += 1
Label2.Text = "Removed: " + found.ToString
TextBox1.Text = TextLine
End If
AppendAllText(String, String)
Opens a file, appends the specified string to the file, and then
closes the file. If the file does not exist, this method creates a
file, writes the specified string to the file, then closes the file.
Source
You are repeatedly opening and closing a file, causing overhead. AppendAllText is a convenience method since it performs several operations in one single call but you can now see why it's not performing well in a big loop.
The fix is easy. Open the file once when you start your loop and close it at the end. Make sure that you always close the file properly even when an exception occurs. For that, you can either invoke the Close in a Finally block, or use a context manager, that is keep your file write operations within a Using block.
And you could remove the print to console as well. Display management has a cost too. Or you could print status updates every 10K lines or so.
When you've done all that, you should notice improved performance.
My Final Code - It works a lot faster now (500mbs in 1 minute)
Using sw As StreamWriter = File.CreateText(output)
For Each oneLine As String In File.ReadLines(FILE_NAME)
Try
If changeme = True Then
changeme = False
GoTo Again2
End If
If oneLine.Contains(":") Then
Dim TestString = oneLine.Substring(0, oneLine.IndexOf(":")).Trim()
Dim TestString2 = oneLine.Substring(oneLine.IndexOf(":")).Trim()
If CheckForAlphaCharacters(TestString) = False And CheckForAlphaCharacters(TestString2) = False Then
sw.WriteLine(oneLine)
Else
found += 1
End If
ElseIf oneLine.Contains(";") Or oneLine.Contains("|") Or oneLine.Contains(" ") Then
Dim oneLineReplac As String = oneLine.Replace(" ", ":")
Dim oneLineReplace As String = oneLineReplac.Replace("|", ":")
Dim oneLineReplaced As String = oneLineReplace.Replace(";", ":")
If oneLineReplaced.Contains(":") Then
Dim TestString3 = oneLineReplaced.Substring(0, oneLineReplaced.IndexOf(":")).Trim()
Dim TestString4 = oneLineReplaced.Substring(oneLineReplaced.IndexOf(":")).Trim()
If CheckForAlphaCharacters(TestString3) = False And CheckForAlphaCharacters(TestString4) = False Then
sw.WriteLine(oneLineReplaced)
Else
found += 1
End If
Else
errors += 1
textstring = oneLine
End If
Else
errors += 1
textstring = oneLine
End If
count += 1
Catch
errors += 1
textstring = oneLine
End Try
Next
End Using
I have a folder with 700+ .jpgs. I also have a Textbox with one filename per line.
I want to check which file does not exist in the folder, but should be there.
This is my code:
Dim Counter As Integer = 0
For Each Line As String In tbFileNames.Lines
Counter = Counter + 1
If (IO.File.Exists(tbFolder.Text & "\" & tbFileNames.Lines(Counter - 1).ToString & ".jpg")) = False Then
tbNotExistingFiles.Text = tbNotExistingFiles.Text & vbNewLine & (tbFileNames.Lines(Counter - 1).ToString)
Else
End If
Next
Problem: I get more than 300 "missing" files, but there should be only 7. When I search for the output filenames, they are in the folder, so the FileExists functions returns false, but it shouldn't.
Where is the problem? Is it the amount of files?
According to this line:
If (IO.File.Exists(tbFolder.Text & "\" & tbFileNames.Lines(Counter - 1).ToString & ".jpg")) = False
Which can be interpreted as:
The tbFolder TextBox contains the directory's path where the images are located.
The tbFileNames TextBox contains the main and complete file names. One file name per line.
Appending the extension & ".jpg" means that the file names in the tbFileNames TextBox are without extensions. And,
You need to get a list of the missing files in that directory and show the result in the tbNotExistingFiles TextBox.
If my interpretation is correct, then you can achieve that using the extension methods like:
Imports System.IO
'...
Dim missingFiles = tbFileNames.Lines.
Select(Function(x) $"{x.ToLower}.jpg").
Except(Directory.GetFiles(tbFolder.Text).
Select(Function(x) Path.GetFileName(x.ToLower)))
tbNotExistingFiles.Text = String.Join(ControlChars.NewLine, missingFiles)
Or by a LINQ query:
Dim missingFiles = From x In tbFileNames.Lines
Where (
Aggregate y In Directory.EnumerateFiles(tbFolder.Text)
Where Path.GetFileName(y).ToLower.Equals($"{x.ToLower}.jpg")
Into Count()
) = 0
Select x
'Alternative to tbNotExistingFiles.Text = ...
tbNotExistingFiles.Lines = missingFiles.ToArray
Note that, there's no need nor use for the File.Exists(..) function in the preceding snippets. Just in case you prefer your approach using For..Loop and File.Exists(..) function, then you can do:
Dim missingFiles As New List(Of String)
For Each line In tbFileNames.Lines
If Not File.Exists(Path.Combine(tbFolder.Text, $"{line}.jpg")) Then
missingFiles.Add(line)
End If
Next
tbNotExistingFiles.Lines = missingFiles.ToArray
I am writing VBA code for an Excel workbook. I would like to be able to open a connection with an Access database, and then import a txt file (pipe delimited) and create a new table in the database from this txt file. I have searched everywhere but to no avail. I have only been able to find VBA code that will accomplish this from within Access itself, rather than from Excel. Please help! Thank you
Google "Open access database from excel VBA" and you'll find lots of resources. Here's the general idea though:
Dim db As Access.Application
Public Sub OpenDB()
Set db = New Access.Application
db.OpenCurrentDatabase "C:\My Documents\db2.mdb"
db.Application.Visible = True
End Sub
You can also use a data access technology like ODBC or ADODB. I'd look into those if you're planning more extensive functionality. Good luck!
I had to do this exact same problem. You have a large problem presented in a small question here, but here is my solution to the hardest hurdle. You first parse each line of the text file into an array:
Function ParseLineEntry(LineEntry As String) As Variant
'Take a text file string and parse it into individual elements in an array.
Dim NumFields As Integer, LastFieldStart As Integer
Dim LineFieldArray() As Variant
Dim i As Long, j As Long
'Determine how many delimitations there are. My data always had the format
'data1|data2|data3|...|dataN|, so there was always at least one field.
NumFields = 0
For I = 1 To Len(LineEntry)
If Mid(LineEntry, i, 1) = "|" Then NumFields = NumFields + 1
Next i
ReDim LineFieldArray(1 To NumFields)
'Parse out each element from the string and assign it into the appropriate array value
LastFieldStart = 1
For i = 1 to NumFields
For j = LastFieldStart To Len(LineEntry)
If Mid(LineEntry, j , 1) = "|" Then
LineFieldArray(i) = Mid(LineEntry, LastFieldStart, j - LastFieldStart)
LastFieldStart = j + 1
Exit For
End If
Next j
Next i
ParseLineEntry = LineFieldArray
End Function
You then use another routine to add the connection in (I am using ADODB). My format for entries was TableName|Field1Value|Field2Value|...|FieldNValue|:
Dim InsertDataCommand as String
'LineArray = array populated by ParseLineEntry
InsertDataCommand = "INSERT INTO " & LineArray(1) & " VALUES ("
For i = 2 To UBound(LineArray)
If i = UBound(LineArray) Then
InsertDataCommand = InsertDataCommand & "'" & LineArray(i) & "'" & ")"
Else
InsertDataCommand = InsertDataCommand & LineArray(i) & ", "
End If
Next i
Just keep in mind that you will have to build some case handling into this. For example, if you have an empty value (e.g. Val1|Val2||Val4) and it is a string, you can enter "" which will already be in the ParseLineEntry array. However, if you are entering this into a number column it will fail on you, you have to insert "Null" instead inside the string. Also, if you are adding any strings with an apostrophe, you will have to change it to a ''. In sum, I had to go through my lines character by character to find these issues, but the concept is demonstrated.
I built the table programmatically too using the same parsing function, but of this .csv format: TableName|Field1Name|Field1Type|Field1Size|...|.
Again, this is a big problem you are tackling, but I hope this answer helps you with the less straight forward parts.
I'm writing a little program that is supposed to rename multiple files in a chosen directory by the filenames.
this is what I use now:
Dim sourcePath As String = dir
Dim searchPattern As String = "*." & ComboBox3.Text
Dim i As Integer = 1
For Each fileName As String In Directory.GetFiles(sourcePath, searchPattern, SearchOption.AllDirectories)
File.Move(Path.Combine(sourcePath, fileName), Path.Combine(sourcePath, type & i & "." & ComboBox3.Text))
i += 1
Next
But i want it to be more like:
For Each fi As FileInfo In Directory.GetFiles(searchPattern).OrderBy(Function(s) s.FullName)
File.Move(Path.Combine(sourcePath, fileName), Path.Combine(sourcePath, type & i & "." & ComboBox3.Text))
i += 1
Next
This is how far I've gotten, but it's not working as i hoped.
Also, I was wondering if it is possible to exclude file-types with the GetFiles, i don't want it to rename text-files.
Thanks :)
Edit:
The first code works almost perfect, it takes the dir from a 'FolderBrowserDialog' and renames all the files within the path-folder. The problem is that it sometimes get the order wrong: lets say i got these 3 files:
01Movie.avi, 07Movie.avi and 11Movie.avi
I want them to be renamed in the order they are in the folder by name:
01Movie.avi should be Movie1.avi, 07Movie.avi -> Movie2.avi and 11Movie.avi -> Movie3.avi
Try not to modify a collection you are still looping through. Created a detatched array to work with. Then sort your new array to get them in the order you described.
Keep in mind that you have SearchOption.AllDirectories, this is going to return subs, so your filename array may not be what you were thinking... Either change the logic of the sort or handle subs separately.
'detached array of file names
Dim fileNames() As String = Directory.GetFiles(sourcePath, searchPattern)
'sort the names System.Array.Sort(Of String)(fileNames)
Dim newFileName As String
For Each fileName As String In fileNames
'manipulate you filename here
'Path.GetFileNameWithoutExtension(fileName) might help
'Path.GetExtension(fileName) might help
newFileName = "newFileNameWithoutExtension" & i.ToString
File.Move(Path.Combine(Path.GetDirectoryName(fileName), fileName), Path.Combine(Path.GetDirectoryName(fileName), newFileName))
i += 1
Next
So I know how to increment. I have the following code working:
Dim startFileName As String = StorageRoot & endFile
Dim endFileName As String = String.Empty
Dim Counter As Integer = 1
Do
Dim myFileInfo As New FileInfo(startFileName)
endFileName = myFileInfo.DirectoryName & IIf(Not myFileInfo.DirectoryName.EndsWith("\"), "\", String.Empty).ToString & _
Path.GetFileNameWithoutExtension(startFileName) & Counter & myFileInfo.Extension
Counter += 1
Loop Until Not IO.File.Exists(endFileName)
endFile = endFileName
This works. But here's my dilemma. Lets say the following files exist:
filename_v1.ext
filename_v2.ext
filename_v3.ext
filename_v4.ext
And then the user deletes one of them, lets say filename_v3.ext
On the next upload with my above logic, I need a way for it to make it v5 and not v3 again. The way it is now, it loops and stops after v2 and creates a v3 again.
Thanks for the help!
You can get an array of all files that match filename_v*.ext from Directory.GetFiles(path, searchPattern).
Dim files As String() = Directory.GetFiles("c:\SomePath", "filename_v*.ext")
Parse out the numbers after _v, order them, then take the highest + 1.