I recently tried Dropnet API to connect dropbox app in my C# project. Everything works fine, but I want to upload large files through chunkupload request.
public void FileUpload()
{
string file = #"E:\threading.pdf";
int chunkSize = 1 * 1024 * 1024;
var buffer = new byte[chunkSize];
int bytesRead;
int chunkCount = 0;
ChunkedUpload chunkupload = null;
using (var fileStream = new FileStream(file, FileMode.Open, FileAccess.Read))
{
while ((bytesRead = fileStream.Read(buffer, 0, buffer.Length)) > 0)
{
chunkCount++;
if (chunkCount == 1)
{
chunkupload = client.StartChunkedUpload(buffer);
}
else
{
chunkupload = client.AppendChunkedUpload(chunkupload, buffer);
}
}
}
var metadata = client.CommitChunkedUpload(chunkupload, "/threading.pdf", true);
}
The file size is 1.6 MB. When i Checked, first chunk contains 1 MB and second one contains 0.6MB but only 13 bytes data gets uploaded in each chunk. Could anyone point out problem here.
Update RestSharp to 104.4.0 to resolve this issue.
There's a problem with RestSharper that is used by Dropnet.
Each uploaded chunk uploads exactly 13 bytes
'System.Byte[]'
The problem is that array of bytes is converted to string using method 'AddParameter'.
I didn't dig too much. I'm trying to use UploadFile method.
Related
I'm trying to read a CR2 file using the Canon EDSDKv0309W. I didn't found an example for this SDK version so I looked at several examples from older versions and created the code below. But I always get EDS_ERR_NOT_SUPPORTED in line EDSDK.EdsGetImage(..).
Using a 32Bit compilation under .Net4.6.1 I can read the correct with and height from images taken with EOS500D and M100. But I don't get the image. So my assumption is I get a wrong pointer from EdsCreateMemoryStream. But I don't see what is wrong and how to debug this.
Any help will be appreciated.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using EDSDKLib;
using System.Drawing;
using System.Drawing.Imaging;
namespace CR2Reader
{
class Program
{
static Bitmap GetImage(IntPtr img_stream, EDSDK.EdsImageSource imageSource)
{
IntPtr stream = IntPtr.Zero;
IntPtr img_ref = IntPtr.Zero;
IntPtr streamPointer = IntPtr.Zero;
EDSDK.EdsImageInfo imageInfo;
uint error = 0;
try
{
//create reference and get image info
error = EDSDK.EdsCreateImageRef(img_stream, out img_ref);
if (error == 0)
{
error = EDSDK.EdsGetImageInfo(img_ref, imageSource, out imageInfo);
if (error == 0)
{
EDSDK.EdsSize outputSize = new EDSDK.EdsSize();
outputSize.width = imageInfo.EffectiveRect.width;
outputSize.height = imageInfo.EffectiveRect.height;
//calculate amount of data
int datalength = outputSize.height * outputSize.width * (int)imageInfo.NumOfComponents * (int)(imageInfo.ComponentDepth / 8);
//create buffer that stores the image
error = EDSDK.EdsCreateMemoryStream((ulong)datalength, out stream);
if (error == 0)
{
//load image into the buffer
error = EDSDK.EdsGetImage(img_ref, imageSource, EDSDK.EdsTargetImageType.RGB16, imageInfo.EffectiveRect, outputSize, stream);
if (error == 0)
{
//make BGR from RGB (System.Drawing (i.e. GDI+) uses BGR)
byte[] buffer = new byte[datalength];
unsafe
{
System.Runtime.InteropServices.Marshal.Copy(stream, buffer, 0, datalength);
byte tmp;
fixed (byte* pix = buffer)
{
for (int i = 0; i < datalength; i += 3)
{
tmp = pix[i]; //Save B value
pix[i] = pix[i + 2]; //Set B value with R value
pix[i + 2] = tmp; //Set R value with B value
}
}
}
//Get pointer to stream
error = EDSDK.EdsGetPointer(stream, out streamPointer);
if (error == 0)
{
//Create bitmap with the data in the buffer
return new Bitmap(outputSize.width, outputSize.height, datalength, PixelFormat.Format24bppRgb, streamPointer);
}
}
}
}
}
return null;
}
finally
{
//Release all data
if (img_ref != IntPtr.Zero) error = EDSDK.EdsRelease(img_ref);
if (stream != IntPtr.Zero) error = EDSDK.EdsRelease(stream);
}
}
static Bitmap ReadCR2Image(string fileName)
{
IntPtr outStream = new IntPtr();
uint error = EDSDK.EdsInitializeSDK();
error += EDSDK.EdsCreateFileStream(fileName,
EDSDK.EdsFileCreateDisposition.OpenExisting,
EDSDK.EdsAccess.Read,
out outStream);
Bitmap bmp = null;
if (error == 0)
{
bmp = GetImage(outStream, EDSDK.EdsImageSource.FullView);
}
if (outStream != IntPtr.Zero)
{
error = EDSDK.EdsRelease(outStream);
}
EDSDK.EdsTerminateSDK();
return bmp;
}
static void Main(string[] args)
{
Bitmap bmp = ReadCR2Image("IMG_3113.CR2");
}
}
}
You are using the wrong EdsImageSource type.
Since you are loading a RAW image you also have to use EdsImageSource.RAWFullView. EdsImageSource.FullView would be appropriate for e.g. a JPG or TIFF.
Once you change that, everything should work just fine.
Edit: just saw you are using RGB16 as target but the rest of the code assumes normal 8bit RGB. You'll have to change a whole bunch of stuff to get that working correctly. I'd advise you to use RGB unless you really need 16bit.
Edit 2: Looks like the library is a bit out of that in that regard (I should really update it). In any case, you can always check the header files of the SDK for up-to-date values. Here's the current definition for EdsImageSource:
enum EdsImageSource
{
FullView = 0,
Thumbnail,
Preview,
RAWThumbnail,
RAWFullView,
}
As for changes required for 16bit:
datalength is incorrect
you're using byte instead of ushort to set the pixels
you create the Bitmap with PixelFormat.Format24bppRgb
then there's another thing where Bitmap doesn't fully support 16Bit. See this article for some in-depth information.
Depending on what you need to do it's probably better to either use the raw pixel data directly as you get it from the SDK or use a different graphics library (e.g. WPF, SkiaSharp, ImageSharp, etc.)
I have a Windows service that periodically uploads files to an FTP server, and will then redownload the file to check that the bytes match exactly. For most files it seems ok, but one of the files we're using recently is a 10mb CSV file and the files are always slightly different (by a couple of bytes).
Below are the upload, download and file compare methods. Note that the Download is setting UseBinary as true but the upload isn't - could this be why?
public string Upload(FileInfo fi, string targetFilename)
{
//copy the file specified to target file: target file can be full path or just filename (uses current dir)
//1. check target
string target;
if (targetFilename.Trim() == "")
{
//Blank target: use source filename & current dir
target = this.CurrentDirectory + fi.Name;
}
else if (targetFilename.Contains("/"))
{
//If contains / treat as a full path
target = AdjustDir(targetFilename);
}
else
{
//otherwise treat as filename only, use current directory
target = CurrentDirectory + targetFilename;
}
string URI = Hostname + target;
//perform copy
System.Net.FtpWebRequest ftp = GetRequest(URI);
//Set request to upload a file in binary
ftp.Method = System.Net.WebRequestMethods.Ftp.UploadFile;
ftp.UseBinary = false;
//Notify FTP of the expected size
ftp.ContentLength = fi.Length;
//create byte array to store: ensure at least 1 byte!
const int BufferSize = 2048;
byte[] content = new byte[BufferSize - 1 + 1];
int dataRead;
string result = null;
//open file for reading
using (FileStream fs = fi.OpenRead())
{
try
{
//open request to send
using (Stream rs = ftp.GetRequestStream())
{
do
{
dataRead = fs.Read(content, 0, BufferSize);
rs.Write(content, 0, dataRead);
} while (!(dataRead < BufferSize));
rs.Close();
}
}
catch (Exception x)
{
result = URI + " - " + x.ToString();
}
finally
{
//ensure file closed
fs.Close();
}
}
ftp = null;
return result;
}
#endregion
public bool Download(string sourceFilename, FileInfo targetFI, bool PermitOverwrite)
{
//1. check target
if (targetFI.Exists && !(PermitOverwrite))
{
throw (new ApplicationException("Target file already exists"));
}
//2. check source
string target;
if (sourceFilename.Trim() == "")
{
throw (new ApplicationException("File not specified"));
}
else if (sourceFilename.Contains("/"))
{
//treat as a full path
target = AdjustDir(sourceFilename);
}
else
{
//treat as filename only, use current directory
target = CurrentDirectory + sourceFilename;
}
string URI = Hostname + target;
//3. perform copy
System.Net.FtpWebRequest ftp = GetRequest(URI);
//Set request to download a file in binary mode
ftp.Method = System.Net.WebRequestMethods.Ftp.DownloadFile;
ftp.UseBinary = true;
ftp.UsePassive = false;
//open request and get response stream
using (FtpWebResponse response = (FtpWebResponse)ftp.GetResponse())
{
using (Stream responseStream = response.GetResponseStream())
{
//loop to read & write to file
using (FileStream fs = targetFI.OpenWrite())
{
try
{
byte[] buffer = new byte[2048];
int read = 0;
do
{
read = responseStream.Read(buffer, 0, buffer.Length);
fs.Write(buffer, 0, read);
} while (!(read == 0));
responseStream.Close();
fs.Flush();
fs.Close();
}
catch (Exception)
{
//catch error and delete file only partially downloaded
fs.Close();
//delete target file as it's incomplete
targetFI.Delete();
throw;
}
}
responseStream.Close();
}
response.Close();
}
return true;
}
Public Function FileCompare(ByVal file1 As String, ByVal file2 As String) As Boolean
' Checks to see if two files are the same
Dim file1byte As Integer
Dim file2byte As Integer
Dim fs1 As FileStream
Dim fs2 As FileStream
If file1 = file2 Then
Return True
End If
fs1 = New FileStream(file1, FileMode.Open)
fs2 = New FileStream(file2, FileMode.Open)
' Simple length test
If fs1.Length <> fs2.Length Then
fs1.Close()
fs2.Close()
Return False
End If
Do
file1byte = fs1.ReadByte()
file2byte = fs2.ReadByte()
Loop While file1byte = file2byte And file1byte <> -1
fs1.Close()
fs2.Close()
Return ((file1byte - file2byte) = 0)
End Function
On certain images, when I call:
PdfImageObject pimg = new PdfImageObject(stream);
Image bmp = pimg.GetDrawingImage();
The Image that is returned is twisted. I've seen this before and it usually has to do with byte alignment but I'm not sure how to get around this.
The /DecodeParms for this object are /EndOfLine true /K 0 /Columns 3300.
I have tried using the GetStreamBytesRaw() with BitMiracle.LibTiff and with it I can get the data formatted properly although the image is rotated. I'd prefer for GetDrawingImage() to decode the data properly if possible, assuming that is the problem.
I could provide the PDF via email if requested.
Thanks,
Darren
For anyone else that runs across this scenario here is my solution. The key to this puzzle was understanding that /K 0 is G3, /K -1 (or anything less than 0) is G4 /K 1 (or anything greater than 0) is G3-2D.
The twisting happens when you try to make G3 compressed data fit into a G4 image which it appears that is what iTextSharp may be doing. I know it definitely does not work with how I have iTextSharp implemented in my project. I confess that I cannot decipher all the decoding stuff that iTextSharp is doing so it could be something I'm missing too.
EndOfLine didn't have any part in this puzzle but I still think putting line feeds in binary data is a strange practice.
99% of this code came from BitMiracle.LibTiff.Net - Thank you.
int nK = 0;// Default to 0 like the PDF Spec
PdfObject oDecodeParms = stream.Get(PdfName.DECODEPARMS);
if (oDecodeParms is PdfDictionary)
{
PdfObject oK0 = ((PdfDictionary)oDecodeParms).Get(PdfName.K);
if (oK0 != null)
nK = ((PdfNumber)oK0).IntValue;
}
using (MemoryStream ms = new MemoryStream())
{
using (Tiff tiff = Tiff.ClientOpen("custom", "w", ms, new TiffStream()))
{
tiff.SetField(TiffTag.IMAGEWIDTH, width);
tiff.SetField(TiffTag.IMAGELENGTH, height);
if (nK == 0 || nK > 0) // 0 = Group 3, > 0 = Group 3 2D
tiff.SetField(TiffTag.COMPRESSION, Compression.CCITTFAX3);
else if (nK < 0) // < 0 = Group 4
tiff.SetField(TiffTag.COMPRESSION, Compression.CCITTFAX4);
tiff.SetField(TiffTag.BITSPERSAMPLE, bpc);
tiff.SetField(TiffTag.SAMPLESPERPIXEL, 1);
tiff.WriteRawStrip(0, rawBytes, rawBytes.Length); //saving the tiff file using the raw bytes retrieved from the PDF.
tiff.Close();
}
TiffStreamForBytes byteStream = new TiffStreamForBytes(ms.ToArray());
using (Tiff input = Tiff.ClientOpen("bytes", "r", null, byteStream))
{
int stride = input.ScanlineSize();
Bitmap result = new Bitmap(width, height, pixelFormat);
ColorPalette palette = result.Palette;
palette.Entries[0] = System.Drawing.Color.White;
palette.Entries[1] = System.Drawing.Color.Black;
result.Palette = palette;
for (int i = 0; i < height; i++)
{
Rectangle imgRect = new Rectangle(0, i, width, 1);
BitmapData imgData = result.LockBits(imgRect, ImageLockMode.WriteOnly, pixelFormat);
byte[] buffer = new byte[stride];
input.ReadScanline(buffer, i);
System.Runtime.InteropServices.Marshal.Copy(buffer, 0, imgData.Scan0, buffer.Length);
result.UnlockBits(imgData);
}
}
}
/// <summary>
/// Custom read-only stream for byte buffer that can be used
/// with Tiff.ClientOpen method.
/// </summary>
public class TiffStreamForBytes : TiffStream
{
private byte[] m_bytes;
private int m_position;
public TiffStreamForBytes(byte[] bytes)
{
m_bytes = bytes;
m_position = 0;
}
public override int Read(object clientData, byte[] buffer, int offset, int count)
{
if ((m_position + count) > m_bytes.Length)
return -1;
Buffer.BlockCopy(m_bytes, m_position, buffer, offset, count);
m_position += count;
return count;
}
public override void Write(object clientData, byte[] buffer, int offset, int count)
{
throw new InvalidOperationException("This stream is read-only");
}
public override long Seek(object clientData, long offset, SeekOrigin origin)
{
switch (origin)
{
case SeekOrigin.Begin:
if (offset > m_bytes.Length)
return -1;
m_position = (int)offset;
return m_position;
case SeekOrigin.Current:
if ((offset + m_position) > m_bytes.Length)
return -1;
m_position += (int)offset;
return m_position;
case SeekOrigin.End:
if ((m_bytes.Length - offset) < 0)
return -1;
m_position = (int)(m_bytes.Length - offset);
return m_position;
}
return -1;
}
public override void Close(object clientData)
{
// nothing to do
return;
}
public override long Size(object clientData)
{
return m_bytes.Length;
}
}
I have been asked to provide a WCF service that allows a blob (potentially 1GB) to be downloaded in chunks as an offset byte[] for consumption by a Silverlight application. Essentially, the operation will have a parameter for number of bytes to offset and the max number of bytes to return, nothing complex I think.
The code I have so far is:
[OperationContract]
public byte[] Download(String url, int blobOffset, int bufferSize)
{
var blob = new CloudBlob(url);
using(var blobStream = blob.OpenRead())
{
var buffer = new byte[bufferSize];
blobStream.Seek(blobOffset, SeekOrigin.Begin);
int numBytesRead = blobStream.Read(buffer, 0, bufferSize);
if (numBytesRead != bufferSize)
{
var trimmedBuffer = new byte[numBytesRead];
Array.Copy(buffer, trimmedBuffer, numBytesRead);
return trimmedBuffer;
}
return buffer;
}
}
I have tested this (albeit with relatively small files < 2MB) and it does work, but my questions are:
Can someone suggest improvements to the code?
Is there a better approach given the requirement?
using (BlobStream blobStream = blob.OpenRead())
{
bool getSuccess = false;
int getTries = 0;
rawBytes = new byte[blobStream.Length];
blobStream.Seek(0, SeekOrigin.Begin);
int blockSize = 4194304; //Start at 4 mb per batch
int index = 0;
int documentSize = rawBytes.Length;
while (getTries <= 10 && !getSuccess)
{
try
{
int batchSize = blockSize;
while (index < documentSize)
{
if ((index + batchSize) > documentSize)
batchSize = documentSize - index;
blobStream.Read(rawBytes, index, batchSize);
index += batchSize;
}
getSuccess = true;
}
catch (Exception e)
{
if (getTries > 9)
throw e;
else
blockSize = blockSize / 2; // Reduce by half for each attempt
}
finally
{ getTries++; }
}
}
You could return the blob as a stream instead of a byte array. There is a code sample in a related question here: Returning Azure BLOB from WCF service as a Stream - Do we need to close it?
Note there are some restrictions on which bindings you can use when you return a stream.
I have a WCF web service that returns a stream. At the client side when i try to read it using below code then i get an exception at line " Byte[] buffer = new Byte[outputMessage.FileByteStream.Length];" saying System.Notsupported. Please advise me on what am i doing wrong here.
FileMetaData metaData = new FileMetaData();
metaData.ProductIDsArray = new string[] { "1", "2" };
metaData.AuthenticationKey = "test";
FileDownloadMessage inputParam = new FileDownloadMessage(metaData);
FileTransferServiceClient obj = new FileTransferServiceClient();
FileDownloadReturnMessage outputMessage = obj.DownloadFile(inputParam);
Byte[] buffer = new Byte[outputMessage.FileByteStream.Length];
int byteRead = outputMessage.FileByteStream.Read(buffer, 0, buffer.Length);
Response.Buffer = false;
Response.Buffer = false;
Response.ContentType = "application/x-zip";
Response.AppendHeader("content-length", buffer.Length.ToString());
Stream outStream = Response.OutputStream;
while (byteRead > 0)
{
outStream.Write(buffer, 0, buffer.Length);
byteRead = outputMessage.FileByteStream.Read(buffer, 0, buffer.Length);
}
outputMessage.FileByteStream.Close();
outStream.Close();
The stream that you're reading from does not support getting the length of the stream (most likely 'cause the length will not be known until the entire file has been downloaded). Read the stream in chunks - similar to how the while loop is doing, but have a fixed size buffer - once you get 0 returned for byteRead you'll know you've hit the end-of-stream.