What would be the best way to access the File object that is contained in the FileBean in Stripes? I am trying to store the file in Amazon's S3 and it requires a byte array. Seems simple enough if I can get to the File object.
FileBean has a getInputStream() method which allows to read every byte from the FileBean. If you really want to store everything in memory in a byte array (which is a bad idea, especially if files can be large), then read evrything from the stream and write it to a ByteArrayOutputStream:
byte[] buffer = new byte[1024];
InputStream in = fileBean.getInputStream();
ByteArrayOutputStream out = new ByteArrayOutputStream();
int read;
while ((read = in.read(buffer)) >= 0) {
out.write(buffer, 0, read);
}
byte[] contentAsByteArray = out.toByteArray();
Related
I have a filename pointing to a text file, including its path, as a string. Now I'd like to load this .csv file into memory stream. How should I do that?
For example, I have this:
Dim filename as string="C:\Users\Desktop\abc.csv"
Dim stream As New MemoryStream(File.ReadAllBytes(filename))
You don't need to load a file into a MemoryStream.
You can simply call File.OpenRead to get a FileStream containing the file.
If you really want the file to be in a MemoryStream, you can call CopyTo to copy the FileStream to a MemoryStream.
I had an XML file being read from disk, using the old XmlReader API. How to read the XML file into memory, and then work with it in memory, instead of reading the disk repeatedly? Based on VB answer from Centro (upvoted) but with a Using block, and in C#.
The key line:
MemoryStream myXMLDocument = new MemoryStream(File.ReadAllBytes(#"c:\temp\myDemoXMLDocument.xml"));
Re the OP's question, if you wanted to load a CSV file into a MemoryStream:
MemoryStream myCSVDataInMemory = new MemoryStream(File.ReadAllBytes(#"C:\Users\Desktop\abc.csv"));
Following is a code snippet showing code to reads through XML document now that it's in a MemoryStream. Basically the same code as when it was coming from a FileStream that pointed to a file on disk. Yes, the XMLTextReader API is old and clunky, but it's what I had to work with in this app.
string myXMLFileName = #"c:\temp\myDemoXMLDocument.xml";
using (MemoryStream myXMLDocument = new MemoryStream(File.ReadAllBytes(myXMLFileName)))
{
myXMLTextReader = new XmlTextReader(myXMLDocument);
myXMLTextReader.WhitespaceHandling = WhitespaceHandling.None;
myXmlTextReader.Read(); // read the XML declaration node, advance to <Batch> tag
while (!myXmlTextReader.EOF)
{
if (myXmlTextReader.Name == "xml" && !myXmlTextReader.IsStartElement()) break;
// advance to <Batch> tag
while (myXmlTextReader.Name == "Batch" && myXmlTextReader.IsStartElement())
{
string BatchIdentifier = myXmlTextReader.GetAttribute("BatchIdentifier");
myXmlTextReader.Read(); // advance to next tag
while (!myXmlTextReader.EOF)
{
if (myXmlTextReader.Name == "Transaction" && myXmlTextReader.IsStartElement())
{
// Start a new set of items
string transactionID = myXmlTextReader.GetAttribute("ID");
myXmlTextReader.Read(); // Read next element, possibly another Transaction tag
}
}
//All Batch tags are completed.Move to next tag
myXmlTextReader.Read();
}
// Close the XML memory stream.
myXmlTextReader.Close();
myXmlDocument.Close();
}
}
You can copy it to a file stream like so:
string fullPath = Path.Combine(filePath, fileName);
FileStream fileStream = new FileStream(fullPath, FileMode.Open);
Image image = Image.FromStream(fileStream);
MemoryStream memoryStream = new MemoryStream();
image.Save(memoryStream, ImageFormat.Jpeg);
//Close File Stream
fileStream.Close();
I am having trouble merging PDFs in-memory. I have 2 memory streams, a master and component stream, the idea is that as each component PDF is built up, the component PDF's bytes are added to the master stream. At the very end of all the components, we have a byte array that's a PDF.
I have the code below, but nothing is copying into my masterStream. I think the issue is with CopyPagesTo, but I'm not familiar enough and the documentation/examples are hard to find.
byte[] updated;
using (MemoryStream masterMemoryStream = new MemoryStream())
{
masterStream.WriteTo(masterMemoryStream);
// Read from master stream (ie. all existing components)
masterMemoryStream.Position = 0;
using (iText.Kernel.Pdf.PdfWriter masterPdfWriter = new iText.Kernel.Pdf.PdfWriter(masterMemoryStream))
using (iText.Kernel.Pdf.PdfDocument masterPdfDocument = new iText.Kernel.Pdf.PdfDocument(masterPdfWriter))
{
using (MemoryStream componentMemoryStream = new MemoryStream())
{
componentStream.WriteTo(componentMemoryStream);
// Read from new component
componentMemoryStream.Position = 0;
using (iText.Kernel.Pdf.PdfReader componentPdfReader = new iText.Kernel.Pdf.PdfReader(componentMemoryStream))
using (iText.Kernel.Pdf.PdfDocument componentPdfDocument = new iText.Kernel.Pdf.PdfDocument(componentPdfReader))
{
// Copy pages from component into master
componentPdfDocument.CopyPagesTo(1, componentPdfDocument.GetNumberOfPages(), masterPdfDocument);
}
}
}
updated = masterMemoryStream.GetBuffer();
}
// Write updates to master stream?
masterStream.SetLength(0);
using (MemoryStream temp = new MemoryStream(updated))
temp.WriteTo(masterStream);
Answer
This is mkl's answer with some of my corrections:
using (MemoryStream temporaryStream = new MemoryStream())
{
masterStream.Position = 0;
componentStream.Position = 0;
using (PdfDocument combinedDocument = new PdfDocument(new PdfReader(masterStream), new PdfWriter(temporaryStream)))
using (PdfDocument componentDocument = new PdfDocument(new PdfReader(componentStream)))
{
componentDocument.CopyPagesTo(1, componentDocument.GetNumberOfPages(), combinedDocument);
}
byte[] temporaryBytes = temporaryStream.ToArray();
masterStream.Position = 0;
masterStream.SetLength(temporaryBytes.Length);
masterStream.Capacity = temporaryBytes.Length;
masterStream.Write(temporaryBytes, 0, temporaryBytes.Length);
}
There are a number of issues in your code. I'll first give you a working version and then go into the issues in your code.
A working version (with an important limitation)
You can combine two PDFs given in MemoryStream instances masterStream and componentStream and get the result in the same MemoryStream instance masterStream as follows:
using (MemoryStream temporaryStream = new MemoryStream())
{
masterStream.Position = 0;
componentStream.Position = 0;
using (PdfDocument combinedDocument = new PdfDocument(new PdfReader(masterStream), new PdfWriter(temporaryStream)))
using (PdfDocument componentDocument = new PdfDocument(new PdfReader(componentStream)))
{
componentDocument.CopyPagesTo(1, componentDocument.GetNumberOfPages(), combinedDocument);
}
byte[] temporaryBytes = temporaryStream.ToArray();
masterStream.Position = 0;
masterStream.Capacity = temporaryBytes.Length;
masterStream.Write(temporaryBytes, 0, temporaryBytes.Length);
masterStream.Position = 0;
}
The limitation is that you have to have instantiated the masterStream with an expandable capacity; the MemoryStream class has a number of constructors only some of which create such an expandable instance while the others create non-resizable instances. For details read here.
Issues in your concept and code
Concatenating PDF files does not result in a valid merged PDF
You describe your concept like this
the idea is that as each component PDF is built up, the component PDF's bytes are added to the master stream
This does not work, though, the PDF format does not allow merging PDFs by simply concatenating them. In particular the (active) objects in a PDF have an identifier number which must be unique in the PDF, concatenating would result in a file with non-unique object identifiers; PDFs contain cross reference structures which map each object identifier to its offset from the file start, concatenating would get all these offsets wrong for the added PDFs; furthermore, a PDF has to have a single root object from which the other objects are referenced directly or indirectly, concatenating would result in multiple root objects.
Writing and immediately overwriting
In your code you have
masterStream.WriteTo(masterMemoryStream);
// Read from master stream (ie. all existing components)
masterMemoryStream.Position = 0;
using (iText.Kernel.Pdf.PdfWriter masterPdfWriter = new iText.Kernel.Pdf.PdfWriter(masterMemoryStream))
Here you write the contents of masterStream to masterMemoryStream, then set the masterMemoryStream position to the start and instantiate a PdfWriter which starts writing there. I.e. your original copy of the masterStream contents get overwritten, surely not what you wanted.
Using MemoryStream.GetBuffer
MemoryStream.GetBuffer does not only return the data written into the MemoryStream by design but the whole buffer; i.e. there may be a lot of trash bytes after the actual PDF in what you retrieve here
updated = masterMemoryStream.GetBuffer();
This may cause PDF processors trying to process your result PDFs to be unable to open the file: PDFs have a pointer to the last cross references at their end, so if you have trash bytes following the actual end of your PDF, PDF processors may not find that pointer.
PS
As worked out in the comments, the code above works fine in case of constantly growing stream lengths (which usually will happen in the use case at hand) but in general one needs to restrict the stream size before writing the new content, e.g. like this:
...
masterStream.Position = 0;
masterStream.SetLength(temporaryBytes.Length); // <<<<
masterStream.Capacity = temporaryBytes.Length;
...
The ADLS .NET SDK has some good examples for reading and creating text files. This uses StreamReader and this shouldn't be used with binary files. I tried using BinaryReader but have been unsuccessful.
https://learn.microsoft.com/en-us/azure/data-lake-store/data-lake-store-data-operations-net-sdk
//Read file contents
using (var readStream = new StreamReader(client.GetReadStream(fileName)))
{
string line;
while ((line = readStream.ReadLine()) != null)
{
Console.WriteLine(line);
}
}
Can the .NET SDK create/read binary? If so, are there any examples of doing this?
Can the .NET SDK create/read binary? If so, are there any examples of doing this?
Short answer is yes, please refer to the following demo code.
Create binary File
AdlsClient adlsClient = AdlsClient.CreateClient($"{datalakeAccount}.azuredatalakestore.net", clientCreds);
using (var stream = adlsClient.CreateFile("file name", IfExists.Overwrite))
{
byte[] textByteArray = File.ReadAllBytes(#"local file path");
stream.Write(textByteArray, 0, textByteArray.Length);
}
read binary file and write to local file
using (var filesream = adlsClient.GetReadStream("1.png"))
{
MemoryStream memorystream = new MemoryStream();
filesream.CopyTo(memorystream);
memorystream.Position = 0;
File.WriteAllBytes(#"filename", memorystream.ToArray());
}
I have a URL to MP4 audio file that I need to send to Speech-To-Text API. The API accepts only WAV stream. I am using NAudio 1.7.3 and the following code to download the file and to get the appropriate stream to be sent to API:
string filePath = "C:\Windows\Temp\file.wav";
using (MediaFoundationReader reader = new MediaFoundationReader(audioFileURL))
{
WaveFileWriter.CreateWaveFile(filePath, reader);
}
System.IO.FileStream fs = new FileStream(filePath, FileMode.Open);
Then I send the fs stream to API and everything works correctly, although very slowly because of I/O to/from disk.
I decided to rewrite this code and execute all required in memory. For this purpose I wrote the following code (that does not provide me a correct stream):
using (MediaFoundationReader reader = new MediaFoundationReader(audioLocation)){
MemoryStream ms = new MemoryStream();
IgnoreDisposeStream ids = new IgnoreDisposeStream(ms);
WaveFileWriter writer = new WaveFileWriter(ids, reader.WaveFormat);
//Doing one of the following (both provide the same outcome):
//1. reader.CopyTo(ids);
//or
//2. this code from NAudio source:
var buffer = new byte[reader.WaveFormat.AverageBytesPerSecond * 4];
while (true)
{
int bytesRead = reader.Read(buffer, 0, buffer.Length);
if (bytesRead == 0)
{
// end of source provider
break;
}
// Write will throw exception if WAV file becomes too large
writer.Write(buffer, 0, bytesRead);
}
writer.Dispose();
Stream streamToSendToAPI = ids.SourceStream;
//Send streamToSendToAPI to Speech-To-Text API
}
My expectation is that using second code example, where I create stream with WAV header and then add the data to the stream, would provide me a valid WAV stream. However, when I send it to speech-to-text API, the API gives error that indicates that stream cannot be processed (meaning that stream is invalid).
Please advise how to fix the in-memory code example to create a valid WAV stream
You need to rewind the memory stream back to the beginning
ms.Position = 0
I've decided to implement encryption for file transfers in my service. File transfers prior to this were not encrypted, and they were sent and received flawlessly with the exact same number of bytes.
Now I've introduced asymmetrical and symmetrical encryption into the mix to encrypt the data as it passes over the TCP protocol. I use asymmetrical to do an initial handshake passing the symmetrical key to the other party encrypted by the asymmetric public key. From then on out, the receiver of the file calls the sender periodically, and the sender generates a new initialization vector, encrypts the data with the symmetric key, and sends it over to be decrypted by the receiver using the IV and same symmetric key.
The chunk size I'm using is 2mb, such that the byte size of the generated chunks, with exception to the last chunk which varies, is 2097152. When AES encrypts this file with PaddingMode.PKCS7 and CipherMode.CBC, the resulting byte size is 2097168. It's gained about 16 bytes during the encryption process.
Now initially I thought this is where my problem was, but when I decrypt the data on the receiving end, it goes back to the 2097152 byte length and I write it to the file. I've proven to myself that it does indeed encrypt and decrypt the data.
On a small enough file, the file sizes from the original to the sender seem to be exactly the same. However, as I step up to larger file sizes, there exists a descrepency. On a video file(Wildlife.wmv from windows 7 install) of size 26,246,026 bytes, I am instead receiving a finished transfer that is of 26,246,218 bytes.
Why is there this size difference? What am I doing wrong here?
Here's some of my code.
For my encryption I am using the following class to encrypt or decrypt, returning a result in the form of a byte array.
public class AesCryptor
{
public byte[] Encrypt(byte[] data, byte[] key, byte[] iv)
{
using (SymmetricAlgorithm aes = new AesManaged())
{
aes.Key = key;
aes.IV = iv;
aes.Padding = PaddingMode.PKCS7;
aes.Mode = CipherMode.CBC;
using (ICryptoTransform encryptor = aes.CreateEncryptor(key, iv))
{
return Crypt(data, key, iv, encryptor);
}
}
}
public byte[] Decrypt(byte[] data, byte[] key, byte[] iv)
{
using (SymmetricAlgorithm aes = new AesManaged())
{
aes.Key = key;
aes.IV = iv;
aes.Padding = PaddingMode.PKCS7;
aes.Mode = CipherMode.CBC;
using (ICryptoTransform decryptor = aes.CreateDecryptor(key, iv))
{
return Crypt(data, key, iv, decryptor);
}
}
}
private byte[] Crypt(byte[] data, byte[] key, byte[] iv, ICryptoTransform transform)
{
using (MemoryStream memoryStream = new MemoryStream())
{
using (CryptoStream cryptoStream = new CryptoStream(memoryStream, transform, CryptoStreamMode.Write))
{
cryptoStream.Write(data, 0, data.Length);
cryptoStream.FlushFinalBlock();
}
return memoryStream.ToArray();
}
}
}
The sender of the file is encrypting the data(after the handshake of the private symmetric key) with this code(and a lot more that doesn't pertain to the actual encryption process. Note the chunkedFile.NextChunk(). This calls a method on the class that is doing the file chunking for me, returning 2mb chunk sizes unless the final size is smaller.
byte[] buffer;
byte[] iv = new byte[symmetricEncryptionBitSize / 8];
using (var rngCrypto = new RNGCryptoServiceProvider())
rngCrypto.GetBytes(iv);
AesCryptor cryptor = new AesCryptor();
buffer = cryptor.Encrypt(chunkedFile.NextChunk(), symmetricPrivateKey, iv);
The code below is what the receiver of the file uses(not all of it, this is what pertains to the decrypting of the data). The data is being written to a file stream(writer).
FileMessage message = hostChannel.ReceiveFile();
moreChunks = message.FileMetaData.MoreChunks;
UpdateTotalBytesTransferred(message);
writer.BaseStream.Position = filePosition;
byte[] decryptedStream;
// Copy the message stream out to a memory stream so we can work on it afterwards.
using (var memoryStream = new MemoryStream())
{
message.ChunkData.CopyTo(memoryStream);
decryptedStream = cryptor.Decrypt(memoryStream.ToArray(), symmetricPrivateKey, message.FileMetaData.InitializationVector);
}
writer.Write(decryptedStream);
By the way, in case it is needed, NextChunk is a very simple method.
public byte[] NextChunk()
{
if (MoreChunks) // If there are more chunks, procede with the next chunking operation, otherwise throw an exception.
{
byte[] buffer;
using (BinaryReader reader = new BinaryReader(File.OpenRead(FilePath)))
{
reader.BaseStream.Position = CurrentPosition;
buffer = reader.ReadBytes((int)MaximumChunkSize);
}
CurrentPosition += buffer.LongLength; // Sets the stream position to be used for the next call.
return buffer;
}
else
throw new InvalidOperationException("The last chunk of the file has already been returned.");
}
EDIT: It seems that for every chunk transferred, and thus every encryption, I am gaining 16bytes in file size. This does not happen with extremely small file sizes.
Well I solved the issue.
It turns out I was sending in the message data the chunkLength of the encrypted chunk data. So for every chunk I sent, even though I decrypted and wrote the correct filedata, I was advancing the stream position by the length of the encrypted data. This means every time I decrypted, when transferring more than 1 chunk(this is why the small files of only 1 chunk size didn't have problems) I was adding 16 bytes to the file size.
People helping me probably wouldn't have been able to figure this out, because I didn't include all of the data in the client side or the server side to see this. But thankfully I managed to answer it myself.
On the sender side, I was creating my FileMessage like this.
FileMessage message = new FileMessage();
message.FileMetaData = new FileMetaData(chunkedFile.MoreChunks, chunkedFile.ChunkLength, chunkedFile.CurrentPosition, iv);
message.ChunkData = new MemoryStream(buffer);
If you see the second parameter of FileMetaData constructor, I'm passing in chunkedFile.ChunkLength which is supposed to be the length of the chunk. I was doing this on the encrypted chunk data, which resulted in sending the incorrect chunk length.
The client on the other hand, was receiving this extra information. If you look near the end, you'll see the code filePosition += message.FileMetaData.ChunkLength;. I was using that erroneous chunkLength to advance the file position. It turns out that setting of the streamPosition was not even necessary.
using (BinaryWriter writer = new BinaryWriter(File.OpenWrite(fileWritePath)))
{
writer.BaseStream.SetLength(0);
while (moreChunks)
{
FileMessage message = hostChannel.ReceiveFile();
moreChunks = message.FileMetaData.MoreChunks;
UpdateTotalBytesTransferred(message);
writer.BaseStream.Position = filePosition;
byte[] decryptedStream;
// Copy the message stream out to a memory stream so we can work on it afterwards.
using (var memoryStream = new MemoryStream())
{
message.ChunkData.CopyTo(memoryStream);
Debug.WriteLine("Received Encrypted buffer Length: " + memoryStream.Length);
decryptedStream = cryptor.Decrypt(memoryStream.ToArray(), symmetricPrivateKey, message.FileMetaData.InitializationVector);
Debug.WriteLine("Received Decrypted buffer Length: " + decryptedStream.Length);
}
writer.Write(decryptedStream);
TotalBytesTransferred = message.FileMetaData.FilePosition;
filePosition += message.FileMetaData.ChunkLength;
}
OnTransferComplete(this, EventArgs.Empty);
StopSession();
}
Such a simple bug, but one that wasn't leaping out at me quickly at all.