We just found out that our WebApi returns trough the WebSocket protocol the message in chunks. These are continuation frames as per RFC6455 specification. While we listen only the first frame is retrieved from Karate and the others lost. This is always a string with 4082 chars length and around 16k bytes size. Is there a way to make Karate listen until the final frame is received so the whole message can be evaluated?
Here is a visualization from Fiddler showing how the frames are received:
Dev of Karate here. We are interested in closing gaps if any, so would you be able to help us get a sample WebSocket endpoint, maybe public or some sample code for us to replicate.
Karate uses Netty as the underlying library. A quick search tells me that it would be easy to support continuation frames if we don't already. Feel free to open a feature request to discuss further.
Related
I know this question was already asked number of times a long time ago,but always remained unanswered.
I have webrtc client which transmits stream trough server(flashphoner) to browser. I need the way to mark specific frames by 4byte label on client side and parse this label in browser using js code.
Other theoretical ability is to add textual/qrcode watermarks and parse it on browser side using some ocr or qrparser library. The problem that i dont know how it possible to access decoded frame data on browser side. Any suggestions?
While something like this hasn't been possible in the past, the WebRTC Insertable Streams / Encoded Transform (specification) API allows this but browser support varies.
https://webrtc.github.io/samples/src/content/insertable-streams/endtoend-encryption/ shows a sample that a trivial XOR encryption and, more important for your use-case, adds a four-byte checksum.
I have a client/server audio synthesizer where the server (java) dynamically generates an audio stream (Ogg/Vorbis) to be rendered by the client using an HTML5 audio element. Users can tweak various parameters and the server immediately alters the output accordingly. Unfortunately the audio element buffers (prefetches) very aggressively so changes made by the user won't be heard until minutes later, literally.
Trying to disable preload has no effect, and apparently this setting is only 'advisory' so there's no guarantee that it's behavior would be consistent across browsers.
I've been reading everything that I can find on WebRTC and the evolving WebAudio API and it seems like all of the pieces I need are there but I don't know if it's possible to connect them up the way I'd like to.
I looked at RTCPeerConnection, it does provide low latency but it brings in a lot of baggage that I don't want or need (STUN, ICE, offer/answer, etc) and currently it seems to only support a limited set of codecs, mostly geared towards voice. Also since the server side is in java I think I'd have to do a lot of work to teach it to 'speak' the various protocols and formats involved.
AudioContext.decodeAudioData works great for a static sample, but not for a stream since it doesn't process the incoming data until it's consumed the entire stream.
What I want is the exact functionality of the audio tag (i.e. HTMLAudioElement) without any buffering. If I could somehow create a MediaStream object that uses the server URL for its input then I could create a MediaStreamAudioSourceNode and send that output to context.destination. This is not very different than what AudioContext.decodeAudioData already does, except that method creates a static buffer, not a stream.
I would like to keep the Ogg/Vorbis compression and eventually use other codecs, but one thing that I may try next is to send raw PCM and build audio buffers on the fly, just as if they were being generated programatically by javascript code. But again, I think all of the parts already exist, and if there's any way to leverage that I would be most thrilled to know about it!
Thanks in advance,
Joe
How are you getting on ? Did you resolve this question ? I am solving a similar challenge. On the browser side I'm using web audio API which has nice ways to render streaming input audio data, and nodejs on the server side using web sockets as the middleware to send the browser streaming PCM buffers.
I'm still trying to master Twisted while in the midst of finishing an application that uses it.
My question is:
My application uses LineReceiver.sendLine to send messages from a Twisted TCP server.
I would like to know if the sendLine succeeded.
I gather that I need to somehow add a success (and error?) callback to sendLine but I don't know how to do this.
Thanks for any pointers / examples
You need to define "succeeded" in order to come up with an answer to this.
All sendLine does immediately (probably) is add some bytes to a send buffer. In some sense, as long as it doesn't raise an exception (eg, MemoryError because your line is too long or TypeError because your line was the number 3 instead of an actual line) it has succeeded.
That's not a very useful kind of success, though. Unfortunately, the useful kind of success is more like "the bytes were added to the send buffer, the send buffer was flushed to the socket, the peer received the bytes, and the receiving application acted on the data in a persistent way".
Nothing in LineReceiver can tell you that all those things happened. The standard solution is to add some kind of acknowledgement to your protocol: when the receiving application has acted on the data, it sends back some bytes that tell the original sender the message has been handled.
You won't get LineReceiver.sendLine to help you much here because all it really knows how to do is send some bytes in a particular format. You need a more complex protocol to handle acknowledgements.
Fortunately, Twisted comes with a few. twisted.protocols.amp is one: it offers remote method calls (complete with responses) as a basic feature. I find that AMP is suitable for a wide range of applications so it's often safe to recommend for new development. It largely supersedes the older twisted.spread (aka "PB") which also provides both remote method calls and remote object references (and is therefore more complex - in my experience, more complex than most applications need). There are also some options that are a bit more standard: for example, Twisted Web includes an HTTP implementation (HTTP, as you may know, is good at request/response style interaction).
I am trying to understand, why does Netty SSL mode work on strange way?
Also, the problem is following, when any SSL client(https browser, java client using ssl, also any ssl client application) connects to Netty server I get on beginning the full message, where I can recognize correctly the protocol used, but as long the channel stays connected, any following messages have strange structure, what is not happening same way with non-ssl mode.
As example on messageReceived method when the https browser connects to my server:
I have used PortUnificationServerHandler to switch protocols.. (without using nettys http handler, it is just example, because i use ssl mode for my own protocol too)
first message is ok, I get full header beginning with GET or POST
than I send response...
second message is only one byte long and contains "G" or "P" only.
third message is than the rest beginning either with ET or OST and the rest of http header and body..
here again follows my response...
fourth message is again one byte long and again contains only one byte..
fifth message again the rest... and on this way the game goes further..
here it is not important, which sub protocol is used, http or any else, after first message I get firstly one byte and on second message the rest of the request..
I wanted to build some art of proxy, get ssl data and send it unencoded on other listener, but when I do it directly without waiting for full data request, the target listener(http server as example) can not handle such data, if the target gets one byte as first only (even if the next message contains the rest), the channel gets immediately closed and request gets abandoned..
Ok, first though would be to do following, cache the first byte temporarily and wait for next message and than join those messages, and only than response, that works fine, but sometimes that is not correct approach, because the one byte is sometimes really the last message byte, and if i cache it and await wrongly next message, i can wait forever, because the https browser expects at this time some response and does not send any data more..
Now the question, is it possible to fix this problem with SSL? May be there are special settings having influence on this behavior?
I want fully joined message at once as is and not firstly first byte and than the rest..
Can you please confirm, that with newer Netty versions you have same behaving by using PortUnificationServerHandler (but without netty http handler, try some own handler.)
Is this behavior Ok so, I do not believe, it was projected so to work..
What you're experiencing is likely to be due to the countermeasures against the BEAST attack.
This isn't a problem. What seems to be the problem is that you're assuming that you're meant to read data in terms of messages/packets. This is not the case: TCP (and TLS/SSL) are meant to be used as streams of continuous data. You should keep reading data while data is available. Where to split incoming data where it's meaningful is guided by the application protocol. For HTTP, the indications are the blank line after the header and the Content-Length or chunked transfer encoding for the entity.
If you define your own protocol, you'll need a similar mechanism, whether you use plain HTTP or SSL/TLS. Assuming you don't need it only works by chance.
I had experienced this issue and found it was caused bu using JDK1.7. Moving back to JDK1.6 solved it. I did not have time to investigate further but have assumed for now that the SSLEngine implementation has changed in the JDK. I will investigate further when time permits.
I need a network framework that can post raw data such as 0x01. or any raw data i need posted to a server. Not just a HTTP request. I can't seem to find any network frameworks like this. Also, If there are none that currently exist. How would i go about writing the sockets? I can't seem to get that down. which is why i am looking for a existing framework. Thanks for any help. I have tried using sockets but, I was unable to convert the CFArrayRef. I tried many things, But, It crashed every time i tested. So, I am running out of options but to use a prebuilt framework.
Note, I do not want to do only a request like:
POST / HTTP/1.1
Host: localhost
Content-Length: 4
0x01
No, I would like to send RAW data so, The following would be sent to the remote server through a socket.
0x01
This would not be visible raw text however, It would be converted to a data string then sent. I know how to convert most of the stuff i need to do, Its the sending and receiving the response I need the framework for or something that'd work. Any tutorials online or examples on how to do something similar that I can modify to fit to my needs would be greatly appreciated.
How do i send raw data to a socket?
Where do i find a framework like this?
Can you provide tutorials that I can modify to fit my needs?
The best framework for this is CocoaAsyncSocket. While it allows sending arbitrary data on the socket, it also provides very useful abstraction of much of the busywork of socket management. Look particularly a the GCD version that replaces the older RunLoop based code.
I've built several low-level protocols on this stack. I recommend it highly.
You can either use raw sockets or the CFStream API. I recommend the latter.