Please do not bother asking why I am wanting to write a 13GB (or more) file to the database - its just the nature of the beast that I have to do this.
My problem is I think its loading it all into memory and I dont want to do that, I need to stream it into the blob either in chunks or on-the-fly using as little memory as possible.
here is the code I currently have:
Using fs As New System.IO.FileStream(InPutFile, System.IO.FileMode.Open, System.IO.FileAccess.Read, System.IO.FileShare.Read)
Using sr As New System.IO.StreamReader(fs)
Dim data(fs.Length - 1) As Byte
fs.Read(data, 0, fs.Length)
Using cmd As New SQLite.SQLiteCommand(TSQLDB)
data = IO.File.ReadAllBytes(InPutFile)
cmd.CommandText = "UPDATE [FileSpace] SET [FileData]=#DAT WHERE [ID]=" & MyID
cmd.Prepare()
cmd.Parameters.Add("#DAT", DbType.Binary, fs.Length)
cmd.Parameters("#DAT").Value = data
cmd.ExecuteNonQuery()
cmd.Dispose()
End Using
data = Nothing
End Using
End Using
Any help would be most appreciated :) Thanks!
I think there might be limits to SQLite - just looking on the sqlite.org website, blobs are limited to 2147483647 bytes, which only equates to approx 2GB. There is a command I have found: SQLITE_LIMIT_LENGTH
I think the only other way around this is going to be for me to insert this is to split up the file into 2GB chunks and insert them that way, then re-join the blobs into one file when saving. Nothing is ever easy! - thanks for the comments though - it gave me food for thought and got me to find this possible resolution.
Related
I have a speed problem and memory efficiency, I'm reading the cutting the big chunk of bytes from the .bin files then writing it in another file, the problem is that to read the file i need to create a huge byte array for it:
Dim data3(endOFfile) As Byte ' end of file is here around 270mb size
Using fs As New FileStream(path, FileMode.Open, FileAccess.Read, FileShare.None)
fs.Seek(startOFfile, SeekOrigin.Begin)
fs.Read(data3, 0, endOFfile)
End Using
Using vFs As New FileStream(Environment.GetFolderPath(Environment.SpecialFolder.Desktop) & "\test.bin", FileMode.Create) 'save
vFs.Write(data3, 0, endOFfile)
End Using
so it takes a long time to procedure, what's the more efficient way to do it?
Can I somehow read and write in the same file stream without using a bytes array?
I've never done it this way but I would think that the Stream.CopyTo method should be the easiest method and as quick as anything.
Using inputStream As New FileStream(...),
outputStream As New FileStream(...)
inputStream.CopyTo(outputStream)
End Using
I'm not sure whether that overload will read all the data in one go or use a default buffer size. If it's the former or you want to specify a buffer size other than the default, there's an overload for that:
inputStream.CopyTo(outputStream, bufferSize)
You can experiment with different buffer sizes to see whether it makes a difference to performance. Smaller is better for memory usage but I would expect bigger to be faster, at least up to a point.
Note that the CopyTo method requires at least .NET Framework 4.0. If you're executing this code on the UI thread, you might like to call CopyToAsync instead, to avoid freezing the UI. The same two overloads are available, plus a third that accepts a CancellationToken. I'm not going to teach you how to use Async/Await here, so research that yourself if you want to go that way. Note that CopyToAsync requires at least .NET Framework 4.5.
I am building a communication library based on the Net.Sockets.TcpClient class. During some unit tests I wanted to test how large a datapacket could be before running into problems. My theory was that the actual size would not matter because the TcpClient would split the data into parts because of its internal sendbuffer. But the actual size did matter because somewhere around 600KB I discovered loss of data.
What my test does is create a local server and a local client that connect with each other. Then it sends a specific (large) package in a loop to test if the server receives it well. I left out all checks but after sending the data a check runs that the data is exactly the same as what was send before looping again. So there is never more data in the pipe than the size that I specified. This code is the client part in my unit test. The server part is a whole library so I cannot post that.
Using myClient As New Net.Sockets.TcpClient()
myClient.BeginConnect("127.0.0.1", ServerPort, Nothing, Nothing)
'Code that checks if the connection has been made
'Create a large string
Dim SendString as String = StrDup(1024000, "A")
Dim SendBytes as Byte() = Text.Encoding.ASCII.GetBytes(SendString)
'Loop the test
For i As Int32 = 1 To 10000
Wait.Reset()
myClient.Client.Send(SendBytes)
Wait.WaitOne(10000) 'Wait for the server to acknowledge
'Run checks to make sure the data is good, otherwise end loop
Next
myClient.Close()
End Using
What happens is that at some random point the server does not receive all data. It looks like sending 1.024.000 bytes of data works most of the times but not always. The iteration at which it fails is random but a loop will never finish 10.000 iterations successfully. I tested the loop with 512.000 bytes and that works. I also tested 600.000 bytes and that failed. I do not know what the actual size is at which it starts failing because it does not seem to be a hard limit. I cannot figure out the problem. Is the TcpClient somehow limited or do I exceed an internal buffer of some kind? I checked the SendBufferSize of the TcpClient and it was 65536. I have no idea if that has anything to do with it. Packages larger than that buffer seem to be sending just fine.
I'm hell bent on making this work with NAudio, so please tell me if there's a way around this. I have streaming raw audio coming in from a serial device, which I'm trying to play through WaveOut.
Attempt 1:
'Constants 8000, 1, 8000 * 1, 1, 8
Dim CustomWaveOutFormat = WaveFormat.CreateCustomFormat(WaveFormatEncoding.Pcm, SampleRate, Channels, AverageBPS, BlockAlign, BitsPerSample)
Dim rawStream = New RawSourceWaveStream(VoicePort.BaseStream, CustomWaveOutFormat)
'Run in background
Dim waveOut = New WaveOut(WaveCallbackInfo.FunctionCallback())
'Play stream
waveOut.Init(rawStream)
waveOut.Play()
This code works, but there's a tiny problem - the actual audio stream isn't raw PCM, it's raw MuLaw. It plays out the companding like a Beethoven's 5th on cheese-grater. If I change the WaveFormat to WaveFormatEncoding.MuLaw, I get a bad format exception because it's raw audio and there are no RIFF headers.
So I moved over to converting it to PCM:
Attempt 2:
Dim reader = New MuLawWaveStream(VoicePort.BaseStream, SampleRate, Channels)
Dim pcmStream = WaveFormatConversionStream.CreatePcmStream(reader)
Dim waveOutStream = New BlockAlignReductionStream(pcmStream)
waveOut.Init(waveOutStream)
Here, CreatePcmStream tries to get the length of the stream (even though CanSeek = false) and fails.
Attempt 3
waveOutStream = New BufferedWaveProvider(WaveFormat.CreateMuLawFormat(SampleRate, Channels))
*add samples when OnDataReceived()*
It too seems to suffer from lack of having a header.
I'm hoping there's something minor I missed in all of this. The device only streams audio when in use, and no data is received otherwise - a case which is handled by (1).
To make attempt (1) work, your RawSourceWaveStream should specify the format that the data really is in. Then just use another WaveFormatConversionStream.CreatePcmStream, taking rawStream as the input:
Dim muLawStream = New RawSourceWaveStream(VoicePort.BaseStream, WaveFormat.CreateMuLawFormat(SampleRate, Channels))
Dim pcmStream = WaveFormatConversionStream.CreatePcmStream(muLawStream);
Attempt (2) is actually very close to working. You just need to make MuLawStream.Length return 0. You don't need it for what you are doing. BlockAlignReductionStream is irrelevant to mu-law as well since mu law block align is 1.
Attempt (3) should work. I don't know what you mean by lack of a header?
In NAudio you are building a pipeline of audio data. Each stage in the pipeline can have a different format. Your audio starts off in Mu-law, then gets converted to PCM, then can be played. A buffered WaveProvider is used for you want playback to continue even though your device has stopped providing audio data.
Edit I should add that the IWaveProvider interface in NAudio is a simplified WaveStream. It has only a format and a Read method, and is useful for situations where Length is unknown and repositioning is not possible.
Dim objBmpImage AS Bitmap = New Bitmap(9450, 6750)
On execution of above line of code application crashes - Argument Exception- Parameter is not valid.
Please advice.
I think this bitmap requires a huge contiguous unmanaged memory to store the bitmap bits.
More than what is available for your process. So its one way of saying the size you want is not supported. Try reducing and it would work.
Dim objBmpImage AS Bitmap = New Bitmap(objYourOriginBmpImage, New Size(9450, 6750))
You are using the name objBmpImage thrice and in the wrong context.
U r using the same object in the bitmap constructor before its actually being created. Try passing an already existing bitmap to the constructor.
You're bitmap is too large to process (I think you will hit a 2GB limit).
Try processing it in smaller chunks and merging files together if necessary.
I used the code below to compress files and they keep growing instead of shrinking. I comressed a 4 kb file and it became 6. That is understandable for a small file because of the compression overhead. I tried a 400 mb file and it became 628 mb after compressing. What is wrong? See the code. (.net 2.0)
Public Sub Compress(ByVal infile As String, ByVal outfile As String)
Dim sourceFile As FileStream = File.OpenRead(inFile)
Dim destFile As FileStream = File.Create(outfile)
Dim compStream As New GZipStream(destFile, CompressionMode.Compress)
Dim myByte As Integer = sourceFile.ReadByte()
While myByte <> -1
compStream.WriteByte(CType(myByte, Byte))
myByte = sourceFile.ReadByte()
End While
sourceFile.Close()
destFile.Close()
End Sub
If the underlying file is itself highly unpredictable (already compressed or largely random) then attempting to compress it will cause the file to become bigger.
Going from 400 to 628Mb sounds highly improbable as an expansion factor since the deflate algorithm (used for GZip) tends towards a maximum expansion factor of 0.03% The overhead of the GZip header should be negligible.
Edit: The 4.0 c# release indicates that the compression libraries have been improved to not cause significant expansion of uncompressable data. This suggests that they were not implementing the "fallback to the raw stream blocks" mode. Try using SharpZipLib's library as a quick test. That should provide you with close to identical performance when the stream is incompressible by deflate. If it does consider moving to that or waiting for the 4.0 release for a more performant BCL implementation. Note that the lack of compression you are getting strongly suggests that there is no point you attempting to compress further anyway
Are you sure that writing byte by byte to the stream is a really good idea? It will certainly not have ideal performance characteristics and maybe that's what confuses the gzip compressing algorithm too.
Also, it might happen that the data you are trying to compress is just not really well-compressable. If I were you I would try your code with a text document of the same size as text documents tend to compress much better than random binary.
Also, you could try using a pure DeflateStream as opposed to a GZipStream as they both use the same compression algorithm (deflate), the only difference is that gzip adds some additional data (like error checking) so a DeflateStream might yield smaller results.
My VB.NET is a bit rusty so I'll rather not try to write a code example in VB.NET. Instead, here's how you should do it in C#, it should be relatively straightforward to translate it to VB.NET for someone with a bit of experience: (or maybe someone who is good at VB.NET could edit my post and translate it to VB.NET)
FileStream sourceFile;
GZipStream compStream;
byte[] buffer = new byte[65536];
int bytesRead = 0;
while (bytesRead = sourceFile.Read(buffer, 0, 65536) > 0)
{
compStream.Write(buffer, 0, bytesRead);
}
This is a known anomaly with the built-in GZipStream (And DeflateStream).
I can think of two workarounds:
use an alternative compressor.
build some logic that examines the size of the "compressed" output and compares it to the size of the input. If larger, chuck the output and just store the data.
DotNetZip includes a "fixed" GZipStream based on a managed port of zlib. (It takes approach #1 from above). The Ionic.Zlib.GZipStream can replace the built-in GZipStream in your apps with a simple namespace swap.
Thank you all for good answers. Earlier on I tried to compress .wmv files and one text file. I changed the code to DeflateStream and it seems to work now. Cheers.