I am working on a VBA macro which will process a CSV file by creating a temporary file.
I am using Microsoft Excel 2010 32-bit. The 32-bit versions have the 2 Gigabyte memory limits.
The size of CSV file that is being processed using macro is > 1GB.
As stated previously, while processing the CSV file we are saving it to a temporary *xls file.
So the total size while processing the CSV file will be the size of CSV file + size of the temp xls file that we are creating during the processing.
The size of both the files together is beyond the limit of 2GB and hence excel is crashing.
We have 2 scenarios:
When the user opens a CSV file using macro, I show a message box to the user if the user opens a CSV file whose size is > 1GB and close the excel.
When the user opens a CSV file using macro, and the CSV file size < 1GB and then user adds some data to it which will make its
size > 1GB, I have to show a message box to the user as soon as the file size reaches 1GB and close the excel without losing any data.
My question,
For the first case, I can use FileLen which returns a Long value specifying the length of a file in bytes.
For the second case: I am not getting how to calculate the file size when user is editing the CSV file using macro.
Need help,
Thanks in advance.
One character = 1 byte. So when user modifies CSV you can track chnges in the following manner:
On cell-value-changed event you add amount of characters in cell + 1 (for comma or any other separator) as amount of bytes to size of file at the beginning. This way you control the size during modifying file.
Keep in mind, if cell already had "something" in it, you have to subtract this value (amount of characters/bytes), as it will be overwritten.
The below code might help you, in the below code i have used 1 MB as reference. you can change it for 1GB or based on your attributes.
Private Sub Worksheet_SelectionChange(ByVal Target As Range)
Call FileSize
End Sub
Sub FileSize()
Dim LResult As Long
strFileFullName = ActiveWorkbook.FullName
LResult = FileLen(strFileFullName)
If LResult > 1000 Then
MsgBox "FileSize is large " & (LResult)
ActiveWorkbook.Close (savechanges)
End If
End Sub
Related
I have to load a textfile into my software, which can be really big (at least 1.5 GB), because I need to read the last line of this file to enumerate some elements with the next part of the script. The needed time can be very long but, sometimes, it is not possible to read the file because of the following error:
System.OutOfMemoryException: 'Array dimensions exceeded supported range.'
Is there a way to solve this issue? Or maybe a different - and better - path that I can follow to do what I need?
EDIT I:
Here follows more details:
I'm generating the aforementioned textfile from a batch-script which is run from the software of mine by pressing a button
Since I need to read a number contained in the last row of the generated textfile, I'm loading the file into the software, the pressure of a button is needed and for the relative Sub I'm using the following command:
Dim path as string
path = "C:\textfile.txt"
RichTextBox1.LoadFile(path, RichTextBoxStreamType.PlainText)
If you only need the last line of the file...
Dim LastLine = File.ReadLines("file.txt").Last()
Ive tried using several methods to save images, one was using binary code. the issue with that is that it saves a few hidden characters in the file name, and then the file becomes funky, where the only program that can open it is mspaint, and when i try to resave the file, it defaults to wanting to save the image as *heic format.
anyone have any ideas on how to save an inlineshape in Word to your desktop.
```
Open strOutFileName For Binary Access Write As #1
i = i + 1
vData = shapeCurrent.Range.EnhMetaFileBits
lWritePos = 1
'put writes to the file that has been created
Put #1, lWritePos, vData
Close #1
```
VBA.
Step 1
The MS Project file is open;
The user starts the macro;
The form is opened;
The user enters the path;
The user clicks "Save";
The user closes the form;
The user closes the MS Project file.
Step 2
The user opens the MS Project file;
The user wins the macro;
The form is opened;
The form displays the path that the user has registered for "Stage 1";
Questions
How to make that when the user opens the form a second time (Step 2) in the form was displayed the path that was saved in (Step 1)?
In other words, after the form was closed (Step 1), the value of the textbox was retained?
Can this textbox value be saved in the MS Project file?
Or should I save it in a separate file?
How is this best done?
Add a custom file property to store information in the MS Project file. For example:
Sub StorePath(newPath As String)
Dim test As String
test = GetPath()
If Len(test) = 0 Then
ActiveProject.CustomDocumentProperties.Add Name:="UserPath", LinkToContent:=False, Type:=msoPropertyTypeString, Value:=newPath
Else
ActiveProject.CustomDocumentProperties("UserPath") = newPath
End If
End Sub
Function GetPath() As String
On Error Resume Next
GetPath = ActiveProject.CustomDocumentProperties("UserPath")
End Function
The information will be stored in the file itself, different files can have different paths stored, and if opened on another computer, the path is still available.
To save a single value on a user's computer, regardless of which file is opened, use SaveSetting and GetSetting, as mentioned by Sam in the comments above. These are not stored with the file and would not be visible on other computers.
A project's task 0 (the project summary task) is rarely used, so its notes field can be a good place to store long data. Unlike ActiveProject.CustomDocumentProperties, task 0's notes isn't constrained by a 255 character limit.
Accessing task 0's notes is a little tricky. On any other task, you'd use
ActiveProject.Tasks(someTaskID).Notes = "really long strings"
'where someTaskID is an integer variable
but task 0's notes are accessed by
ActiveProject.Comments = "really long strings"
Further to Rachel's answer and in response to Jerred S.' comment, It is easy to overcome the 255 char limit of CustomDocumentProperties and to store War and Peace in there. Write a function such as function storeMyCDPstring(CDPNames as string, CDPVal as string). It will need to chop CDPVal into not-to-exceed 255 character packets and store these as indexed CustomDocumentProperties. Example, you want to store a 1000 char string in CustomDocumentProperty named "MyCDP". You adopt an arbitary naming convention - CDPs will be indexed by "#~#-n":
chars 1 to 255 will be stored as CustomDocumentProperty "myCDP#~#-1",
chars 256 to 510 will be stored as CustomDocumentProperty "myCDP#~#-2",
chars 511 to 765 will be stored as CustomDocumentProperty "myCDP#~#-3",
chars 766 to 1000 will be stored as CustomDocumentProperty "myCDP#~#-4"
You will need to write a function such as function getMyCDPstring(CDPNames as string) which must retrieve, concatenate and return all the substrings. You need also a managed delMyCDP function that will delete all the packets.
Question, if anyone could help please: The file I am reading from inPath is very large 300MB to 1 GB +. I need to load the file into the variable wholeFile as shown in the below program. Approximately 200 MB files works fine but larger files bomb out (Out of Memory Exception Error). The purpose is once file is loaded into the variable, I would need to run RegEx and pick certain section of the file and save somewhere else. Thanks once again for your kind attention.
Dim inPath As String = "C:\temp\300MB-File.txt"
Dim outPath As String = "C:\temp\myFileNew2.txt"
Dim wholeFile as String = ""
Using sw As StreamWriter = File.CreateText(outPath)
For Each oneLine As String In File.ReadLines(inPath)
sw.WriteLine(oneLine)
wholeFile = wholeFile & vbCrLf & oneLine
Next
End Using
The way you're doing that is abominable. Why would you read a file line by line if your purpose is to store the entire contents in a single variable? Why wouldn't you load the whole file in one go?
Dim fileContents = File.ReadAllText(filePath)
That may still have memory issues with large files but the way you're doing will use exponentially more memory. Each time you do that concatenation to the String, you create a new String object and copy the previous contents into it along with the new text. That means that, for a file with N lines, you are going to create N Strings. The first will contain the first line, then the second will contain the first two lines, then the third will contain the first three lines, etc, etc.
If you really want to read the file line by line then you could use a StringBuilder, which avoids so much memory reallocation. Even better would be to get the size of the file first and then create the StringBuilder with the appropriate capacity from the get go, so no reallocation would be needed at all.
When you get right down to it though, files of that size are going to be an issue no matter what. You will either need to ensure that enough memory is allocated to your app to handle it or else you'll have to break the file up into chunks and process each chunk separately. If your regex won't match very large portions of the file then you can simply make each chunk overlap by a line or two and then handle the special cases where you get duplicate matches in the overlapping section.
I am trying to make some code that will generate a random number and then check numbers on each line in a text file to see if has already been generated. I have everything but code that will check for the number generated in the text file. Any ideas?
Here is the code I have so far:
Dim Rlo As New IO.StreamReader("C:\Users\Somebody\Documents\Visual Studio 2012\Projects\RobloxRecruitV1\RobloxRecruitV1\bin\Debug\" & TheFileName.Text & ".txt")
Dim firstLine As String
'read first line
firstLine = Rlo.ReadLine()
'read secondline
TheText.Text = Rlo.ReadLine()
rndnumber = New Random
number = rndnumber.Next(firstLine, TheText.Text)
TextBox1.Text = number.ToString
I can't give you the exact code (It's been a long time since I did anything in VB6...)
but....
I can tell you that using a stream reader is the wrong approach.
A stream reader is exactly what it's name suggests. A constant stream of data, it starts and then stops when it reaches an end.
Now while it's true that you can to a small extent seek back and forth in a stream, that's not really what you need in this case.
What you need is to load all the lines of your file into an in memory array or some kind of hash table, then your task simply becomes one of looking to see if a given index exists.
If you have no choice but to use the file as is on disk (Due to size restrictions for example) then the approach you need is this:
1) Open the file
2) Set you position to the beginning
3) enter a loop reading sequential lines
4) once you have the line that corresponds to the count your looking for close the file and end
5) loop back round until no more lines left
6) close the file
opening and closing, then resetting each time is important, this is so that you KNOW EXACTLY where in the file your starting from each time, you could in theory keep the file open and just reset the position, but that in my mind could be dangerous esp if you have other processes writing to it.
If your file is not very big, then I'd opt for an in memory approach, load the file, perform operations on the in memory array of lines, then save it before exit.