I have a streamed WCF service. In one operation, I receive a file, for upload purposes.
If I try to do something like this
request.FileContent.Length
Then I receive an OperationNotSupported exception. That's Ok.
But how could I get the file size without actually transfering it entirely?
I know I could send this information along with the call, as a Header, but I don't want to go this way.
If WCF is able to limit the request size trough maxReceiveMessageSize. How can I use the same information to check the message/stream size?
In general, you can't know the size of a byte stream without reading it all and counting the bytes, unless there is data at the start of the stream which tells you how many bytes there are in the entire stream, or some other out-of-band way to communicate the length of the stream, such as in the WCF message headers. You will have to go with the Header approach if you want to know the size without reading the stream.
The WCF maxReceiveMessageSize works by counting the bytes as they are received and throwing an exception if the limit is exceeded... it doesn't know the stream length either, and can't pre-emptively prevent the message being received without first reading the maximum allowed number of bytes.
But how could I get the file size without actually transfering it entirely? I know I could send this information along with the call, as a Header, but I don't want to go this way.
You're going to have to send the size of the byte stream down the pipe first there is no other way. (if there was some inbuilt way thats all it would be doing anyway)
It doesn't add much complexity to prepend it to the stream:
var bytes = File.ReadAllBytes("somefile.txt");
stream.Write(BitConverter.GetBytes((Int32)bytes.Length), 0, 4);
stream.Write(bytes, 0, bytes.Length);
and then on the other side when reading the stream:
byte[] fileLengthBytes =new byte[4];
stream.Read(fileLengthBytes, 0, 4);
int length = BitConverter.ToInt32(fileLengthBytes, 0);
//you know the size of the file now, log it or show the user
var fileBytes = new byte[length];
stream.Read(fileBytes, 0, fileBytes.Length);
this is only an example - you may not want to create a byte[] buffer if your stream is large.
Related
I am wondering one thing; How can I serialize multiple objects to a byte array. My goal is to send the serialized object over tcp, receive it, then deserialize it, and recreate it.
My concept is:
The first thing in the byte array will be the "Packet Header" -- This will tell the receiver what type of packet it is; "Chat Message", "File Transfer", etc etc. Then after the header I will add the packet itself. Then at last there will be an "EOF Header" (This will tell the server if the whole packet is received). - The headers are enumerables(as Byte).
where you get these errors would be helpful (essential even) but it is probably related to this:
Public Shared Function Deserialize(Data As Byte()) As Packet
Dim MS As New MemoryStream(Data)
Dim BF As New BinaryFormatter
MS.Position = 0
' or
'MS.Seek(0, SeekOrigin.Begin)
Return DirectCast(BF.Deserialize(MS), Packet)
End Function
After seeding the memstream, the stream position is left at the end. You need to reset it so the BF can read all the bytes. (and you really dont need things like BOF and EOF in the serialized data - even if you are sending multiple things, if you put them in a list, they will either de/serialize in toto or not).
Also look at ProtoBuf-NET - much faster serializer making much smaller packets, and it will let you deserialize into a different assembly-culture-class which NET's BF does not do without basically tricking it.
I'm using CocoaAsyncSocket for an iOS project. I'm trying to read VarInts through an asynchronous interface. The problem is unlike something else like a String, where I can prefix a length, I don't know the length of a varint beforehand. It needs to be processed one byte at a time, but since each read operation is asynchronous other read calls may have been queued in between.
I considered reading into a buffer then processing it, say reading 5 bytes (the max length for a varint-32), and pushing extra bytes back, but that may hang unnecessarily if the varint is only 4 bytes and I'm waiting for a 5th byte to be available.
How can I do this? Also, I cannot change the protocol on the other end, to use fixed size ints.
Here's a snippet of code as Josh requested
- (void)readByte:(void (^)(int8_t))onComplete {
NSUInteger size = 1;
int32_t tag = OSAtomicAdd32(1, &_nextTag);
dispatch_async(self.dispatchQueue, ^{
[self.onCompleteHandlers setObject:(^void (NSData* data) {
int8_t x = 0;
[data getBytes:&x length:size];
onComplete(x);
}) forKey:[NSNumber numberWithInteger:((NSInteger) tag)]];
[self.socket readDataToLength:size withTimeout:-1 tag:tag];
});
}
A callback is saved in a dictionary, which is used in the delegate method socket: didReadData: withTag.
Suppose I'm reading a VarInt byte by byte:
execute read first byte for varint
don't know if we need to read another byte for a varint or not; that depends on the result of the first read
(possible) read another byte for something else
read second byte for varint, but now it's actually the 3rd byte being read
I can imagine using a flag to indicate whether or not I'm in a multipart-read, and a queue to hold reads that should be executed after the multipart-read, and I've started writing it but it's quite messy. Just wondering if there is a standard/recommended/better way to approach this problem.
in short there are 4 ways to know how much to read from a socket...
read some format that you can infer the length from like the Content-Length header... only works if the whole request can be put together before the body is sent.
read until some pattern: like \r\n\r\n at the end of the headers
read until some timeout... after you get no bytes after n seconds you flush the buffers and close the connection.
read until the server closes the connection... actually used to be pretty common.
these each have problems and I would probably lean in your case from using some existing protocol.
of course there is overhead to doing it that way, and you may find that you don't want to use any of that application level stuff and your requests may be like:
client>"doMath(2+5)\0"
server>"(7)\0"
but it is hard to answer your general question specifically.
edit:
So I looked into the varint base-128 issue a little more and I think really only a timeout or the server closing the connection will work, if you are writing these right at the TCP level which is horrible...
I want both the client and server to write and read resp. at a constant rate (which can be configured on the GUI of the client) to the buffer.
Say,
I am able to send from the client at 150 bytes per packet
Now, I should be able to read also at 150 bytes per packet on the server too
Since, both are connected through a socket, can we retrieve the socket params (set on the client size, like 150 here) from the server end, using the tcpServer object.
Or is it must to send an initial setup packet which tells about these client params and so accordingly the server can continue?
It's kinda usual to sort message sizes out at the application level. You could indeed send a 'setup message' as the first data after a successful connect. You should send this setup message in a form that will not be misunderstood due to endianness or the number of bytes received per read call. Perhaps a fixed-size messge in ASCII, maybe five bytes:
'00150'
The server can then read five bytes only, convert to integer, save it in the server-client socket object so that the server always knows how many bytes to send and then issue a read call for that number of bytes.
Alternatively, you could use a simple protocol that embeds the size into each message, eg:
SOH
"0"
"0"
"1"
"5"
"0"
[150 bytes of data]
EOT
Rgds,
Martin
I'm hell bent on making this work with NAudio, so please tell me if there's a way around this. I have streaming raw audio coming in from a serial device, which I'm trying to play through WaveOut.
Attempt 1:
'Constants 8000, 1, 8000 * 1, 1, 8
Dim CustomWaveOutFormat = WaveFormat.CreateCustomFormat(WaveFormatEncoding.Pcm, SampleRate, Channels, AverageBPS, BlockAlign, BitsPerSample)
Dim rawStream = New RawSourceWaveStream(VoicePort.BaseStream, CustomWaveOutFormat)
'Run in background
Dim waveOut = New WaveOut(WaveCallbackInfo.FunctionCallback())
'Play stream
waveOut.Init(rawStream)
waveOut.Play()
This code works, but there's a tiny problem - the actual audio stream isn't raw PCM, it's raw MuLaw. It plays out the companding like a Beethoven's 5th on cheese-grater. If I change the WaveFormat to WaveFormatEncoding.MuLaw, I get a bad format exception because it's raw audio and there are no RIFF headers.
So I moved over to converting it to PCM:
Attempt 2:
Dim reader = New MuLawWaveStream(VoicePort.BaseStream, SampleRate, Channels)
Dim pcmStream = WaveFormatConversionStream.CreatePcmStream(reader)
Dim waveOutStream = New BlockAlignReductionStream(pcmStream)
waveOut.Init(waveOutStream)
Here, CreatePcmStream tries to get the length of the stream (even though CanSeek = false) and fails.
Attempt 3
waveOutStream = New BufferedWaveProvider(WaveFormat.CreateMuLawFormat(SampleRate, Channels))
*add samples when OnDataReceived()*
It too seems to suffer from lack of having a header.
I'm hoping there's something minor I missed in all of this. The device only streams audio when in use, and no data is received otherwise - a case which is handled by (1).
To make attempt (1) work, your RawSourceWaveStream should specify the format that the data really is in. Then just use another WaveFormatConversionStream.CreatePcmStream, taking rawStream as the input:
Dim muLawStream = New RawSourceWaveStream(VoicePort.BaseStream, WaveFormat.CreateMuLawFormat(SampleRate, Channels))
Dim pcmStream = WaveFormatConversionStream.CreatePcmStream(muLawStream);
Attempt (2) is actually very close to working. You just need to make MuLawStream.Length return 0. You don't need it for what you are doing. BlockAlignReductionStream is irrelevant to mu-law as well since mu law block align is 1.
Attempt (3) should work. I don't know what you mean by lack of a header?
In NAudio you are building a pipeline of audio data. Each stage in the pipeline can have a different format. Your audio starts off in Mu-law, then gets converted to PCM, then can be played. A buffered WaveProvider is used for you want playback to continue even though your device has stopped providing audio data.
Edit I should add that the IWaveProvider interface in NAudio is a simplified WaveStream. It has only a format and a Read method, and is useful for situations where Length is unknown and repositioning is not possible.
How to publish a stream using librtmp library?
I read the librtmp man page and for publishing , RTMP_Write() is used.
I am doing like this.
//Code
//Init RTMP code
RTMP *r;
char uri[]="rtmp://localhost:1935/live/desktop";
r= RTMP_Alloc();
RTMP_Init(r);
RTMP_SetupURL(r, (char*)uri);
RTMP_EnableWrite(r);
RTMP_Connect(r, NULL);
RTMP_ConnectStream(r,0);
Then to respond to ping/other messages from server, I am using a thread to respond like following:
//Thread
While (ThreadIsRunning && RTMP_IsConnected(r) && RTMP_ReadPacket(r, &packet))
{
if (RTMPPacket_IsReady(&packet))
{
if (!packet.m_nBodySize)
continue;
RTMP_ClientPacket(r, &packet); //This takes care of handling ping/other messages
RTMPPacket_Free(&packet);
}
}
After this I am stuck at how to use RTMP_Write() to publish a file to Wowza media server?
In my own experience, streaming video data to an RTMP server is actually pretty simple on the librtmp side. The tricky part is to correctly packetize video/audio data and read it at the correct rate.
Assuming you are using FLV video files, as long as you can correctly isolate each tag in the file and send each one using one RTMP_Write call, you don't even need to handle incoming packets.
The tricky part is to understand how FLV files are made.
The official specification is available here: http://www.adobe.com/devnet/f4v.html
First, there's a header, that is made of 9 bytes. This header must not be sent to the server, but only read through in order to make sure the file is really FLV.
Then there is a stream of tags. Each tag has a 11 bytes header that contains the tag type (video/audio/metadata), the body length, and the tag's timestamp, among other things.
The tag header can be described using this structure:
typedef struct __flv_tag {
uint8 type;
uint24_be body_length; /* in bytes, total tag size minus 11 */
uint24_be timestamp; /* milli-seconds */
uint8 timestamp_extended; /* timestamp extension */
uint24_be stream_id; /* reserved, must be "\0\0\0" */
/* body comes next */
} flv_tag;
The body length and timestamp are presented as 24-bit big endian integers, with a supplementary byte to extend the timestamp to 32 bits if necessary (that's approximatively around the 4 hours mark).
Once you have read the tag header, you can read the body itself as you now know its length (body_length).
After that there is a 32-bit big endian integer value that contains the complete length of the tag (11 bytes + body_length).
You must write the tag header + body + previous tag size in one RTMP_Write call (else it won't play).
Also, be careful to send packets at the nominal frame rate of the video, else playback will suffer greatly.
I have written a complete FLV file demuxer as part of my GPL project FLVmeta that you can use as reference.
In fact, RTMP_Write() seems to require that you already have the RTMP packet formed in buf.
RTMPPacket *pkt = &r->m_write;
...
pkt->m_packetType = *buf++;
So, you cannot just push the flv data there - you need to separate it to packets first.
There is a nice function, RTMP_ReadPacket(), but it reads from the network socket.
I have the same problem as you, hope to have a solution soon.
Edit:
There are certain bugs in RTMP_Write(). I've made a patch and now it works. I'm going to publish that.