I have a string array that is populated with thousands of entries. I am wanting the quickest way to save this data to disk and to also load it back up.
Currently I am looping through the array and appending this data to a string and then saving this string. This takes a long time.
What is the fastest/most efficient way to do this?
Thanks
Write
Dim FileName as string=Application.StartupPath & "\myarray.txt"
IO.File.WriteAllLines(FileName,myarray)
Read
Dim FileName as string=Application.StartupPath & "\myarray.txt"
Dim myarray() As String = File.ReadAllLines(FileName)
How important is it that the resulting file be human-readable? If it's mostly there to be read later by the program, you should definitely use serialization:
public void SaveArray (string[] array)
{
NetDataContractSerializer serializer = new NetDataContractSerializer();
using (FileStream file = System.IO.File.Create(_filename))
{
serializer.Serialize(file, array);
}
}
Edit:
To summarize the advantages of this method over the other two suggested below, the first being using a StringBuilder to concatenate a big string and then saving it to disk, and the second using WriteAllLines to write the string array to disk:
The first method will completely lose fidelity to the original list of strings, since it will glob them together into one string. Can you separate them later? Are there well-known delimiters between them? Not necessarily. It's more efficient than just calling + between them, but it's not a good way to store them.
The second method is better, since each string will be saved to a different line, and can be read back with ReadAllLines, but you're still on shaky ground here. What happens if any string contains the NewLine character? It will be one string when written, but two different strings when read back. Again, calling WriteAllLines/ReadAllLines relies on newlines being good delimiters, when that's not necessarily the case.
Using a serializer - and it doesn't matter if it's the NetDataContractSerializer, BinaryFormatter or any other - will maintain full fidelity with your original data structure, with only a little bit of overhead.
Try using StringBuilder
See the msdn reference for more
Why stringbuilder?
When dealing with large files. It is advisible to use the StringBuilder class rather than the regular string class.
Appending strings together, requires the old string to be copied into a new string.
String Builders are buffers that can change size easily and you can append and delete faster.
You should loop through your data using a string builder which is much faster than ordinary string.
then create a stream writer/ reader (when you want to load the file) from IO library and create the file on your disk.
Related
Sorry if the problem is so basic, I'm a bit used to python not VB.net
I'm trying to read text file data (numbers) and store it in array/list
# Sample of text
1.30e+03,1.30e+03,1.30e+03
5.4600e+02,2.7700e+02,2.8000e+02
# PS: I can control the output of the numbers to have delimiter = ',' or space between numbers, whatever is easier to import
I wrote the following code to read string data and store it. yet, I don't know how to have a multidimensional array (2D or 3D) instead of 1D string (e.g. for the text above, it would be 2x3 array)
' Import Data
Comp_path = FinalPath & "Components_colors.txt"
reader = New StreamReader(Comp_path)
Dim W As String = ""
Dim wArray(10) As String
Dim i As Integer = 0
Do Until reader.Peek = -1
W = reader.ReadLine()
wArray(i) = W
i += 1
Loop
Moreover, I don't know the length of the text file, so I can't determine the length of the array like I did in the code above for the string wArray
For a file like this, you should turn to NuGet for a dedicated CSV parser. There are parsers built into .NET you could also use, but pulling one off of NuGet will also let you parse the values directly into something other than a string.
But if you really don't want to do that you can start with this (assuming Option Infer):
Public Function ImportData(filePath As String) As IEnumerable(Of Double())
Dim lines = File.ReadLines(filePath)
Return lines.Select(Function(line) line.Split(",").Select(AddressOf Double.Parse).ToArray())
End Function
And use it like this:
Comp_path = FinalPath & "Components_colors.txt"
Dim result = ImportData(Comp_path)
Note this code doesn't actually do any meaningful work yet. It doesn't even read the file. What it does is give you an object (result) that you can use with a For Each loop or linq operations. It will read the file in a just-in-time way, parsing out the data for each line as it goes. If you want an array (or List, which you should use in .Net more often), you can append a ToList() call to the end:
Comp_path = FinalPath & "Components_colors.txt"
Dim result = ImportData(Comp_path).ToList()
But you should try to avoid doing that it. It's much less efficient in terms of memory use. The first sample will only ever need to keep one line of the file in memory at a time. Adding ToArray() or ToList() needs to load the entire file.
Some more notes:
Many newer dynamic platforms like Python don't actually use real arrays in the formal computer science sense (fixed block of contiguous memory). Rather, they use collections, and just call them arrays. .Net has collections, too, but when you declare an array, you get an array. This has nice benefits for performance, but if you don't know you want that or how to take advantage of it you're probably better off asking for a generic List most of the time instead.
Thanks to cultural/internationalization issues, parsing numeric (or date) values to string and back again is much slower and more error-prone than you've believed in the past, especially coming from a dynamic platform. It is slow on these other platforms, too, but they want you to pretend it isn't. The first introduction to a strongly-typed platform like .Net can feel stifling in this area, but once you understand the performance and and reliability benefits, you won't want to go back.
In strongly-typed platforms it is very important to understand the data types you are working with at every level of an expression. Otherwise, building and reading statements like the Return line in my answer will be way more difficult and frustrating than it needs to be.
Does anyone know of a method that allows you to search a string through a text file using StreamReader that allows you to account for multiple instances of finding the results. Basically I am creating a booking application and each time a customer books a seat, their PrimaryKey, FirstName, LastName and the co-ordinates of the seat on a data grid (which I have used as a method to book seats) are generated then saved to a text file.
I want the ability to be able to read multiple instances of a PrimaryKey then find the seat co-ordinates of each line that this PrimaryKey is listed on and repopulate another similar datagridview with these co-ordinates which is all going to be driven by a combobox index change.
It seems a bit complicated to understand but if anyone can help then please let me know.
I just need the knowhow of how to search multiple instances, so after its found the string once then look through the rest of the file to find another instance, I can do the rest by myself.
I'm coding using Visual Basic.Net
Yes it's possible to search multiple times through a file, but you'd either have to reopen the file, or rewind the stream (FileStream.Seek).
Wildly inefficient though.
If it has to remain an unsorted and unstructured file, build an in memory index to it.
If your key is an integer, create a Dictionary<int,int> of Key and Position in the stream.
Then when you want find key X you use FileStream.Seek to move to it, and read a line to get the data. If you find yourself grouping by say aeroplaneID, build a Dictionary<Int, List<Int,Int>>
where the key is the aeroplane id and the list is a list of primary keys and positions in the file.
You could push all that off to a background thread. You could try and get really clever and build them up as you need them. Personally though I'd be trying to move my storage to a more suitable format. You aren't struggling to do this because you've missed a class, you are struggling because you shouldn't.
Something like
Dictionary<int, int> _fileIndex = new Dictionary<int,int>();
using(FileStream fs = new FileStream(DataFileName,FileMode.Open,FileAccess.Read))
{
StreamReader reader = new StreamReader(fs);
int lastPosition = 0;
string currentLine = null;
while(currentLine = reader.ReadLine() != null)
{
String[] data = currentLine.Split(new char[] {','});
int key = int.Parse(data[0]);
fileIndex.Add(key,lastPosition);
lastPosition = fs.Position;
}
}
NB didn't test the above and there should be a bit more error checking in it. If there's alot of data in the line, then might be better off not suing split and just pulling out everything up to the correct delimiter. Also be careful how many indexes you keep live, wouldn't be long before they used up more space than just reading the entire thing in to memory.
Then you could create a class or structure to implement a line in the file, and write a bit of code
to use FileStream.Seek) to get there. If you wanted to load up a bunch of 'em it would make sense to get your list of positions of each one in the file and then sort them in order, then you could rip through the file in it's 'order' picking them out.
I am making an app that lets the user draw on the screen in different colors and brush sizes. I am storing the info about each drawn path in a JSON file once it has been drawn to keep it out of memory. Right now I have it parsing all existing paths, then adding the new one in and writing it all back out again. I want it to simply append the new data into the JSON file without having to read it in and parse it first, that will make it so only one path is ever in memory at a time.
I am using SBJSON, the JSONWriter has a few append functions but I think you need to have the JSON string to append it to first, not the file, meaning I would have to read in the file anyway. Is there a way to do this without reading in the file at all? I know exactly how the data is structured.
It's possible, but you have to cheat a little. You can just create a stand-alone JSON document per path, and append that to the file. So you'll have something like this in your file:
{"name":"path1", "from": [0,3], "to":[3, 9]}
{"name":"path2", "from": [0,3], "to":[3, 9]}
{"name":"path3", "from": [0,3], "to":[3, 9]}
Note that this is not ONE JSON document but THREE. Handily, however, SBJsonStreamParser supports reading multiple JSON documents in one go. Set the supportMultipleDocuments property and plug it into a SBJsonStreamParserAdapter, and off you go. This also has the benefit that if you have many, many paths in your file as you can start drawing before you're finished reading the whole file. (Because you get a callback for each path.)
You can see some information on the use case here.
I'm pretty sure its not possible...what I ended up doing was reading in the JSON file as a string then instead of wasting memory changing all that into Dictionaries and Arrays, I just looked for an instance of part of the string (ex: i wanted to insert something before the string "], "texts"" showed up) where I wanted to insert data and inserted it there and wrote it back out to file.
As far as I can tell this is the best solution.
I have some VB6 code that needs to be migrated to VB.NET, and I wanted to inquire about this line of code, and see if there is a way to implement it in .NET
Dim strChar1 As String * 1
Intellisense keeps telling me that an end of statement is expected.
That's known as a "fixed-length" string. There isn't an exact equivalent in VB.NET.
Edit: Well, OK, there's VBFixedStringAttribute, but I'm pretty sure that exists solely so that automated migration tools can more easily convert VB6 code to VB.NET for you, and it's not really the ".NET way" to do things. Also see the warnings in the article for details on why this still isn't exactly the same thing as a fixed-length string in VB6.
Generally, fixed-length strings are only used in VB6 if you are reading fixed-size records from a file or over the network (i.e. parsing headers in a protocol frame).
For example, you might have a file that contains a set of fixed-length records that all have the format (integer, 1-character-string, double), which you could represent in VB6 as a user-defined type:
Public Type Record
anInteger As Integer
aSingleCharacter As String * 1
aDouble As Double
End Type
This way, VB6 code that reads from the file containing records in this format can read each fixed-sized record stored in the file, and in particular, it will only read 1 byte for aSingleCharacter. Without the * 1, VB6 would have no idea how many characters to read from the file, since a String can normally have any number of characters.
In VB.NET, you can do one of the following, depending on your needs:
If the length matters (i.e. you need to read exactly one byte from some data source, for example) consider using an array instead, such as
Dim aSingleByteArray(1) As Byte
Alternatively, you could use one of the Stream classes. In particular, if you are reading data from a network socket or a file, consider using NetworkStream or FileStream, respectively. A Stream is meant for byte-for-byte access (i.e. raw binary access). StreamReader is a related class that simplifies reading data when it is text-based, so that might be good if you are reading a text file, for example. Otherwise (if dealing with binary data), stick with one of the Stream classes.
If the length doesn't matter, you could just use a "normal" String. That is to say:
Dim aNormalString As String
Which answer is "correct" really depends on why it was declared that way in the original VB6 code.
The fixed length strings has been deprecated in VB.NET because there are several better options.
Since your fixed length string is just one character long, you can use the Char type in this case, as Mark suggested.
Dim strChar1 As Char
Seeing as you're doing a VB6 migration, I'd definitely consider VBFixedStringAttribute as well as the other options listed by Mike Spross, but, in this case, because it is a single character, a Char may be an option in this case too.
As mentioned elsewhere VBFixedString is only acknowledged by the Get and Put VB I/O API. So the best solution (other than rewriting your code that references the "fixed length string") is to write your own equivalent of Microsoft.VisualBasic.Compatibility.VB6.FixedLengthString. See this answer for more details.
VBFixedStringAttribute Class
<VBFixedString(1)> Dim strChar1 As String
ALthough this question was asked ages ago, VB.NET actually has a native fixed-length string -- <VbFixedArray(9)> Public fxdString As Char() 'declare 10-char fixed array. Doing this with scalars actually creates VB6-style Static Arrays.
I have a byte array of a file and I need to save it into my database in a field that has been set aside of type image.
However I have a problem my data access class takes a sql string and commits it to the database for example.
"EXECUTE stored proc #parm1, #parm2, #parm3"
However the problem is I cannot figure out how to transfer the byte array to string so that I can add it as an argument.
I hope this make sense.
I also understand that I can build parameters in com objects but I do not want to do this as it will disrupt my whole data access class and I am not prepared to do this at the moment.
Thanks for any help.
In SQL statements, you can use the hexadecimal notation "0x1323235..." to represent binary data but it's not really a good way to deal with it. You should be using parameters:
sqlCmd.Parameters.AddWithValue("#parameterName", byteArrayInstance)
Answering question of how to burn byte array into a string.
Convert the byte array to a string
byte[] b = new byte[100];
string s = System.Text.ASCIIEncoding.ASCII.GetString(b);