I have read that internal implementation of StringBuilder has changed significantly in .NET4.0:
StringBuilder growing over 85k and moving to LOH?
How does StringBuilder's capacity change?
What is the implication of these changes for specifying capacity during SB creation?
Is it still of benefit to do this?
Dim BigString As New System.Text.StringBuilder(500000)
For Each ...
BigString.Append("some text")
Next
Related
I have a speed problem and memory efficiency, I'm reading the cutting the big chunk of bytes from the .bin files then writing it in another file, the problem is that to read the file i need to create a huge byte array for it:
Dim data3(endOFfile) As Byte ' end of file is here around 270mb size
Using fs As New FileStream(path, FileMode.Open, FileAccess.Read, FileShare.None)
fs.Seek(startOFfile, SeekOrigin.Begin)
fs.Read(data3, 0, endOFfile)
End Using
Using vFs As New FileStream(Environment.GetFolderPath(Environment.SpecialFolder.Desktop) & "\test.bin", FileMode.Create) 'save
vFs.Write(data3, 0, endOFfile)
End Using
so it takes a long time to procedure, what's the more efficient way to do it?
Can I somehow read and write in the same file stream without using a bytes array?
I've never done it this way but I would think that the Stream.CopyTo method should be the easiest method and as quick as anything.
Using inputStream As New FileStream(...),
outputStream As New FileStream(...)
inputStream.CopyTo(outputStream)
End Using
I'm not sure whether that overload will read all the data in one go or use a default buffer size. If it's the former or you want to specify a buffer size other than the default, there's an overload for that:
inputStream.CopyTo(outputStream, bufferSize)
You can experiment with different buffer sizes to see whether it makes a difference to performance. Smaller is better for memory usage but I would expect bigger to be faster, at least up to a point.
Note that the CopyTo method requires at least .NET Framework 4.0. If you're executing this code on the UI thread, you might like to call CopyToAsync instead, to avoid freezing the UI. The same two overloads are available, plus a third that accepts a CancellationToken. I'm not going to teach you how to use Async/Await here, so research that yourself if you want to go that way. Note that CopyToAsync requires at least .NET Framework 4.5.
Please do not bother asking why I am wanting to write a 13GB (or more) file to the database - its just the nature of the beast that I have to do this.
My problem is I think its loading it all into memory and I dont want to do that, I need to stream it into the blob either in chunks or on-the-fly using as little memory as possible.
here is the code I currently have:
Using fs As New System.IO.FileStream(InPutFile, System.IO.FileMode.Open, System.IO.FileAccess.Read, System.IO.FileShare.Read)
Using sr As New System.IO.StreamReader(fs)
Dim data(fs.Length - 1) As Byte
fs.Read(data, 0, fs.Length)
Using cmd As New SQLite.SQLiteCommand(TSQLDB)
data = IO.File.ReadAllBytes(InPutFile)
cmd.CommandText = "UPDATE [FileSpace] SET [FileData]=#DAT WHERE [ID]=" & MyID
cmd.Prepare()
cmd.Parameters.Add("#DAT", DbType.Binary, fs.Length)
cmd.Parameters("#DAT").Value = data
cmd.ExecuteNonQuery()
cmd.Dispose()
End Using
data = Nothing
End Using
End Using
Any help would be most appreciated :) Thanks!
I think there might be limits to SQLite - just looking on the sqlite.org website, blobs are limited to 2147483647 bytes, which only equates to approx 2GB. There is a command I have found: SQLITE_LIMIT_LENGTH
I think the only other way around this is going to be for me to insert this is to split up the file into 2GB chunks and insert them that way, then re-join the blobs into one file when saving. Nothing is ever easy! - thanks for the comments though - it gave me food for thought and got me to find this possible resolution.
I am trying to create city distances array CityDistances(CityId1, CityId2) = Distance
As I don't know how many citys there would be, I need an unlimited array. I tried creating it via Dim CityDistances(,) As Double, but when I try to use it, it throws an exception. How do I achieve that?
Instead of an array, a possibly better alternative (and a little more OOP) is through the use of a List(Of Type).
Dim d As List(Of DIstances) = New List(Of Distances)()
d.Add(New Distances() With {.CityID1=1, .CityID2=2, .Distance=12.4})
d.Add(New Distances() With {.CityID1=1, .CityID2=3, .Distance=15.1})
d.Add(New Distances() With {.CityID1=1, .CityID2=4, .Distance=2.4})
d.Add(New Distances() With {.CityID1=1, .CityID2=5, .Distance=130.1})
Public Class Distances
Public CityID1 as Integer
Public CityID2 as Integer
Public Distance as Double
End Class
This has the advantage to let your list grow without specifying any initial limits.
First of all, by doing this:
Dim CityDistances(,) As Double
You declare a pointer to a two dimensional array, without performing any memory allocation for its elements. It is expected that some method will return this array, and you will use it via this pointer. If you try to use it AS IS, without assigning anything to it, you will get index-out-of-bounds exception, which is normal.
Second, there is no such thing as unlimited arrays. You need to be using list of lists, dictionary of dictionaries, a DataTable or similar, if you want automatic memory management. If you want to stick with arrays, for performance/convenience reasons, you can ReDim it when required (increase dimensions), and preserve contents, like this:
Dim CityDistances(,) As Double
ReDim Preserve CityDistances(10,10)
'.... code goes here ....
ReDim Preserve CityDistances(100,100)
Make sure you know when to ReDim, because every time you do it, .NET will create another array and copy all elements there. As the size of your array grows, it may become a performance factor.
Third, based on the nature of your question, you may want to look into custom implementations of the Matrix class. Here are some links I found through Google, hope you find them useful. Those are C#, but there are free online converters to VB.NET on the internet.
Lightweight fast matrix class in C# (Strassen algorithm, LU decomposition) (free)
Efficient Matrix Programming in C# (code-project, so free as well)
High performance matrix algebra for .NET programming! (paid, 99$ and up)
I'd like to reduce the message size when sending serialized integer over the network.
In the below section buff.Length is 256 - to great an overhead to be efficient!
How it can be reduced to the minimum (4 bytes + minimum overhead)?
int val = RollDice(6);
// Should 'memoryStream' be allocated each time!
MemoryStream memoryStream = new MemoryStream();
BinaryFormatter formatter = new BinaryFormatter();
formatter.Serialize(memoryStream, val);
byte[] buff = memoryStream.GetBuffer();
Thanks in advance,
--- KostaZ
Have a look at protobuf.net...it is a very good serialization lib (you can get it on NuGet). Also, ideally you should be using a "using" statement around your memory stream.
To respond to the comment below, then the most efficient method depends on your use case. If you know exactly what you need to serialize and don't need a general purpose serializer then you could write your own binary formatter, which might have no overhead at all (there is some detail here custom formatters).
This link has a comparison of the BinaryFormatter and protobuf.net for your reference.
Dim objBmpImage AS Bitmap = New Bitmap(9450, 6750)
On execution of above line of code application crashes - Argument Exception- Parameter is not valid.
Please advice.
I think this bitmap requires a huge contiguous unmanaged memory to store the bitmap bits.
More than what is available for your process. So its one way of saying the size you want is not supported. Try reducing and it would work.
Dim objBmpImage AS Bitmap = New Bitmap(objYourOriginBmpImage, New Size(9450, 6750))
You are using the name objBmpImage thrice and in the wrong context.
U r using the same object in the bitmap constructor before its actually being created. Try passing an already existing bitmap to the constructor.
You're bitmap is too large to process (I think you will hit a 2GB limit).
Try processing it in smaller chunks and merging files together if necessary.