I'm facing what seems a charset issue of play when decompressing gzip content from rest services. When I try to run the code snippet below, an error is thrown, saying "Malformed JSON. Illegal character ((CTRL-CHAR, code 31))":
val url:String = "https://api.stackexchange.com/2.0/info?site=stackoverflow"
Async {
WS.url(url)
.withHeaders("Accept-Encoding" -> "gzip, deflate")
.get()
.map { response =>
Ok("Response: " + (response.json \ "items"))
}
}
At first I thought it would be a problem in StackExchange API itself, but I tried a similar service, which uses gzip compression as well, and the same error happens. It's hard to fix the code because I don't even know where is the "Illegal character". Is there something missing or it's actually a bug in play?
The clue I can provide is that the first byte of a gzip stream is 31 (0x1f). So you probably need to do something else to cause the gzip stream to be decompressed.
By the way, I recommend that you not accept deflate encoding, just gzip.
Here is how it can be done with Play 2.3
// set Http compression: https://www.playframework.com/documentation/2.3.x/ScalaWS
val clientConfig = new DefaultWSClientConfig()
val secureDefaults: AsyncHttpClientConfig = new NingAsyncHttpClientConfigBuilder(clientConfig).build()
val builder = new AsyncHttpClientConfig.Builder(secureDefaults)
builder.setCompressionEnabled(true)
val secureDefaultsWithSpecificOptions: AsyncHttpClientConfig = builder.build()
implicit val implicitClient = new NingWSClient(secureDefaultsWithSpecificOptions)
val response = WS.clientUrl("http://host/endpoint/item").withHeaders(("Accepts-encoding", "gzip")).get()
Related
I have functions in Python for compression and decompression of a string (bytearray):
def compress_message(data):
compressor = zlib.compressobj(-1, zlib.DEFLATED, 31, 8, zlib.Z_DEFAULT_STRATEGY)
compressed_data = compressor.compress(data)
compressed_data += compressor.flush()
return compressed_data
def decompress_message(compressed_data):
return zlib.decompress(compressed_data, wbits=31)
I need to convert these functions to kotlin, so I can use them in my mobile app. So far I tried this:
fun String.zlibCompress(): ByteArray {
val input = this.toByteArray(charset("UTF-8"))
val output = ByteArray(input.size * 4)
val compressor = Deflater().apply {
setLevel(-1)
setInput(input)
finish()
}
val compressedDataLength: Int = compressor.deflate(output)
return output.copyOfRange(0, compressedDataLength)
}
However, it gives totally different results for example for string abcdefghijklmnouprstuvwxyz:
Python: 1f8b080000000000000a4b4c4a4e494d4bcfc8cccacec9cdcb2f2d282a2e292d2bafa8ac0200c197b2d21a000000
Kotlin: 789c4b4c4a4e494d4bcfc8cccacec9cdcb2f2d282a2e292d2bafa8ac020090b30b24
Is there any way, how can I modify the kotlin code, so it gives same result as Python?
Thanks for your replies. <3
The 31 parameter in the Python code is requesting a gzip stream, not a zlib stream. In Kotlin, you are generating a zlib stream. (zlib is described in RFC 1950, gzip in RFC 1952.)
It does not appear that Java's Deflater (spelled wrong) class has that option. It does have a nowrap option that gives raw deflate compressed data, around which you can construct your own gzip wrapper, using the RFC to see how.
By the way, the results are not "totally different". You have gzip and zlib wrappers around exactly the same raw deflate compressed data: 4b4c...0020.
I've seen other posts on this, but none seem to help. The issue seems to be adding the parameter to the request. Note this is a .svc -- not wsdl. The C# .svc file has a data contract of:
[DataContract]
public class MyMethod_In
{
[DataMember]
public string rc;
}
Kotlin code: (constants take out for easier reading, and names changed to protect the innocent)
val soapObject = SoapObject("https://qa.mysite.com/ws/MyService.svc/", "MyMethod")
soapObject.addProperty("rc", "xyz")
val envelope = SoapSerializationEnvelope(SoapEnvelope.VER11)
envelope.setOutputSoapObject(soapObject)
envelope.dotNet = true
val httpTransport = HttpTransportSE("https://qa.mysite.com/ws/MyService.svc/MyMethod")
try {
httpTransport.call("https://qa.mysite.com/ws/MyService.svc/MyMethod", envelope)
blah blah
}
catch (e: Exception) {
exception happens with 400 error
}
The problem seems to be the soapObject.addProperty line. I've also tried other code where I construct the property info and set the name, type, and value, but I still always get the 400 error. I tried that with a type of String and a type of JSON, and it made no difference.
I'm not totally sure of some of the hard coded values, but I seem to be calling the service correctly, because I tried another service with no input parameters, and this worked fine and got the response.
I'm probably missing one small thing, but can't find it. Any ideas on why the bad request? Thanks
Figured it out for okhttp3, version 4.2.0. As I suspected, I need to pass JSON data. This code does it:
val json = "{ \"rc\":\"xyz\" }"
val mediaTypeJson = "application/json; charset=utf-8".toMediaType()
val requestBody = json.toString().toRequestBody(mediaTypeJson)
val request = Request.Builder()
.url("https://qa.mysite.com/ws/MyService.svc/MyMethod")
.post(requestBody)
.build()
Should probably clean it up and use gson or something, but at least this works.
I am trying to call a twitter endpoint that gives you a constant streams of json results back to the client
https://documenter.getpostman.com/view/9956214/T1LMiT5U#977c147d-0462-4553-adfa-d7a1fe59c3ec
I try to make a call to the endpoint like this
val url = "https://api.twitter.com/2/tweets/search/stream"
_streamChannel = _client.get<ByteReadChannel>(token, url) //Stops here
val byteBufferSize = 1024
val byteBuffer = ByteArray(byteBufferSize)
_streamChannel?.let {
while (_streamChannel!!.availableForRead > 0) {
_streamChannel!!.readAvailable(byteBuffer, 0, byteBufferSize)
val s = String(byteBuffer)
parseStreamResponseString(s).forEach {
emit(Response.Success(it))
}
}
}
my client.get code is this
suspend inline fun <reified T> get(authKey: String, url: String): T? {
val response = _client.get<HttpResponse>(url) {
header("Authorization", "Bearer $authKey")
}
when (response.status.value) {
in 300..399 -> throw RedirectResponseException(response)
in 400..499 -> throw ClientRequestException(response)
in 500..599 -> throw ServerResponseException(response)
}
if (response.status.value >= 600) {
throw ResponseException(response)
}
return response.receive<T>()
}
When I make the request it just sits there in what I am assuming is waiting for the full response to be returned before giving it to me.
Edit
I also tried using scoped streaming but it just sits at the line readAvailable I know there are messages coming through because when I run the request via cURL I am constantly getting data
_client.get<HttpStatement> {
header("Authorization", "Bearer $authKey")
url(urlString)
contentType(ContentType.Application.Json)
method = HttpMethod.Get
}.execute {
val streamChannel = it.receive<ByteReadChannel>()
val byteBufferSize = 1024
val byteBuffer = ByteArray(byteBufferSize)
streamChannel.readAvailable(byteBuffer, 0, byteBufferSize) // Stops here
val s = String(byteBuffer)
}
How do I process a constant stream of json data using Ktor?
As far as I am aware, the Ktor client does note expose access to the IO buffer of the request in the way that twitter's streaming API requires.
From the twitter documentation here:
Some HTTP client libraries only return the response body after the connection has been closed by the server. These clients will not work for accessing the Streaming API. You must use an HTTP client that will return response data incrementally. Most robust HTTP client libraries will provide this functionality. The Apache HttpClient will handle this use case, for example.
What you are doing is telling Ktor that the thing you are getting is a ByteReadChannel, and so, once the request closes (which will never happen with this twitter endpoint) the Ktor client would attempt to use whatever plugin (json for example) you were using to parse that data into a ByteReadChannel. It would also not be able to do that, because the data you are getting from twitter is not a ByteReadChannel, it is a new line seperated list of json objects.
I am trying to send a text via get request to my server, but I have some struggles with spaces/special characters.
How can I encode my text for the request?
(And how to decode it in php?)
Here is my code:
var app = Application.currentApplication();
app.includeStandardAdditions = true;
var text = app.displayDialog("enter your text:", { defaultAnswer: "" }).textReturned;
var result = JSON.parse(app.doShellScript('curl https://example.com?text=' + text));
result
Well, it looks like I found the answer myself:
encodeURIComponent(text)
seems to be working for me. PHP decoding not necessary.
I have a self-hosted WCF REST/webHttpBinding-endpoint-bound service. I have a few streams of different content types that it serves. The content itself is delivered correctly, but any OutgoingResponse.ContentType setting seems to be ignored and instead delivered as "application/xml" every time.
Browsers seems to get over it for javascript and html (depending on how it's to be consumed), but not for css files which are interpreted more strictly. CSS files are how I realized the problem but it's a problem for all Streams. Chromebug and IE developer tools both show "application/xml" regardless of what I put in the serving code for a content type. I've also tried setting the content type header as a Header in OutgoingResponse but that makes no difference and it probably just a long way of doing what OutgoingResponse.ContentType does already.
[OperationBehavior]
System.IO.Stream IContentChannel.Code_js()
{
WebOperationContext.Current.OutgoingResponse.ContentType = "text/javascript;charset=utf-8";
var ms = new System.IO.MemoryStream();
using (var sw = new System.IO.StreamWriter(ms, Encoding.UTF8, 512, true))
{
sw.Write(Resources.code_js);
sw.Flush();
}
ms.Position = 0;
return ms;
}
This behavior is added:
var whb = new WebHttpBehavior
{
DefaultBodyStyle = System.ServiceModel.Web.WebMessageBodyStyle.WrappedRequest,
DefaultOutgoingRequestFormat = System.ServiceModel.Web.WebMessageFormat.Json,
DefaultOutgoingResponseFormat = System.ServiceModel.Web.WebMessageFormat.Json,
HelpEnabled = false
};
I've tried setting AutomaticFormatSelectionEnabled = true and false just in case because it came up in google searches on this issue, but that has no effect on this.
I'm finding enough articles that show Stream and ContentType working together to confuse the heck out of me as to why this isn't working. I believe that the Stream is only intended to be the body of the response, not the entire envelope.
My .svclog doesn't show anything interesting/relevant that I recognize.
============
I can confirm in Fiddler2 that the headers are being delivered as shown in the browser.
...
Content-Type: application/xml; charset=utf-8
Server: Microsoft-HTTPAPI/2.0
...
Solved!
I had something like the following in a MessageInspector:
HttpResponseMessageProperty responseProperty = new HttpResponseMessageProperty();
responseProperty.Headers.Add("Access-Control-Allow-Origin", "*");
reply.Properties["httpResponse"] = responseProperty;
and this was overwriting the already-present HttpResponseMessageProperty in reply.Properties, including any contentType settings. Instead, I tryget the HttpResponseMessageProperty first and use the existing one if found.
I lucked out seeing that one.