return webClient//
.post()//
.uri(whatever.com)
.header("Authorization", "Bearer " + authToken)//
.header("userId", CLIENT_ID)//
.header("clientRequestId", requestId)//
.bodyValue(bodyValue())//
.retrieve()//
.bodyToMono(Responseclass.class)//
.block();
The above is working. But let's say I'm debugging and I just want to dump the raw response json into a String. How would I do that? toString() after retrieve doesn't work, and bodyToMono(String.class) didn't seem to work either. Either way it just printed the default toString value of the address of the pointer.
With Spring Webflux WebClient you can get response as String like this:
WebClient client = WebClient.create("http://someurl.de/something");
String responseBody = client.get().retrieve().toEntity(String.class)
.block().getBody();
In my German blog I wrote an article about using WebClient, there you will find more details around the code snippet above:
https://agile-coding.blogspot.com/2021/01/reactive-webclient.html
Related
I'm using ktor 1.5.3 HTTP client and wondering how can I deserialize HTTP response headers into a list of LinkHeader values. I have the following value in my code:
response.headers.getAll("Link")
which is a list of strings, and I want to get a value of type
List<LinkHeader>
UPDATED:
The details of my use-case:
I have a backend that uses the following response headers to manage pagination:
Link: <https://hostname/v2/issues?orderBy=updated&orderAsc=false&perPage=15>; rel="first"
Link: <https://hostname/v2/issues?orderBy=updated&orderAsc=false&page=2&perPage=15>; rel="prev"
Link: <https://hostname/v2/issues?orderBy=updated&orderAsc=false&page=4&perPage=15>; rel="next"
Link: <https://hostname/v2/issues?orderBy=updated&orderAsc=false&page=116922&perPage=15>; rel="last"
I just have to parse them to understand where is the last page
Since there is not such functionality in Ktor right now, I've created this feature request to address your problem. As a workaround, you can use regular expressions for your particular case to parse headers' values:
data class Link(val url: Url, val rel: String)
fun parse(value: String): Link {
val matches = Regex("""<(.+?)>;\s*rel="(.+?)"""").matchEntire(value) ?: throw Exception("Cannot parse Link header value $value")
val (_, urlString, rel) = (matches.groupValues)
return Link(URLBuilder(urlString).build(), rel)
}
As there is no accurate solution from Ktor, I've implemented a workaround from this article. The same do-while loop worked in my case as well. It makes a redundant API call for an empty last page but works.
I run a web service where I convert a file from one file format into another. The conversion logic is already functioning but now, I want to query this logic via Jersey. Whenever file upload via Jersey is addressed in tutorials / questions, people describe how to do this using multipart form data. I do however simply want to send and return a single file and skip the overhead of sending multiple parts. (The webservice is triggered by another machine which I control so there is no HTML form involved.)
My question is how would I achieve something like the following:
#POST
#Path("{sessionId"}
#Consumes("image/png")
#Produces("application/pdf")
public Response put(#PathParam("sessionId") String sessionId,
#WhatToPutHere InputStream uploadedFileStream) {
return BusinessLogic.convert(uploadedFile); // returns StreamingOutput - works!
}
How do I get hold of the uploadedFileStream (It should be some annotation, I guess which is of course not #WhatToPutHere). I figured out how to directly return a file via StreamingOutput.
Thanks for any help!
You do not have to put anything in the second param of the function; just leave it un-annoted.
The only thing you have to be carefull is to "name" the resource:
The resource should have an URI like: someSite/someRESTEndPoint/myResourceId so the function should be:
#POST
#Path("{myResourceId}")
#Consumes("image/png")
#Produces("application/pdf")
public Response put(#PathParam("myResourceId") String myResourceId,
InputStream uploadedFileStream) {
return BusinessLogic.convert(uploadedFileStream);
}
If you want to use some kind of SessionID, I'd prefer to use a Header Param... something like:
#POST
#Path("{myResourceId}")
#Consumes("image/png")
#Produces("application/pdf")
public Response put(#HeaderParam("sessionId") String sessionId,
#PathParam("myResourceId") String myResourceId,
InputStream uploadedFileStream) {
return BusinessLogic.convert(uploadedFileStream);
}
I'm using VB.NET to communicate with Magento via the API SOAP v1. I had it working fine until I got to a call that needed an associative array. I've tried dictionaries, hashtables, custom types, etc. I did read hashtables and dictionaries are not serializable. So I tried the custom type/object/class. Below is the error I received:
The type XXXX was not expected. Use the Xmlinclude or SoapInclude attribute to specify types that are not known statically.
So I've read a lot of posts in regards to the above error and I can't get anything to work. The error seems at least to say what I'm trying is possible if I do it right, but maybe that is not the case. I don't know much about SOAP, but I can see this never working since the web service might not know how to handle the object.
So my question is if it's possible to send a custom object to the Magento API. Or more broadly is it possible to get SOAP v1 to work with VB.NET. I know v2 is an option but I'm pretty familiar with v1 and already having it working in VB.NET other than this scenario.
Answering a question you didn't ask — but have you considered using the XML-RPC adapter for the V1 API? It exposes the same exact methods as the SOAP API, but since XML-RPC doesn't have the same strong concept of types that SOAP does you may be able to get a generic object through from .NET (said without being that familiar with .NET SOAP or XML-RPC libraries)
It took a lot of digging, but I have it all working. To start I used the PHP SoapClient and noted how it formed associative arrays in the soap request. You can trace requests and responses, pretty handy. From there I wrote my own soap client in VB.NET using a WebRequest object. In doing so I have full control over the xml being sent to the API.
Again the reason I went down this road is ultimately V2 was not working for me. For some reason not all parameters were making it to the API. That and the fact I'm pretty comfortable with V1 too. I've written several custom APIs.
I apologize for the brevity, but there is a lot that went into this. Probably most of my time was hitting multiple dead ends. If anyone wants specifics feel free email me.
EDIT:
Here is the php code I used to see how I needed to format the requests:
$client = new SoapClient('http://www.site.com/index.php/api/soap/?wsdl',array('trace' => TRUE));
$session = $client->login('user','api-key');
echo $client->__getLastRequest() ."\n\n";
echo $client->__getLastRequestHeaders() ."\n\n";
echo $client->__getLastResponse() ."\n\n";
echo $client->__getLastResponseHeaders() ."\n\n";
$result = $client->call($session, 'cataloginventory_stock_item.list','393');
echo $client->__getLastRequest() ."\n\n";
echo $client->__getLastRequestHeaders() ."\n\n";
var_dump($result);
$client->endSession($session);
Below is how to send the request using VB.NET. You will need to construct the XML/SOAP body using the php above as a guide. I made a class per API call which output the needed XML. You will need System.Net, System.Xml & System.IO. I used getSoapHeader() because there is some common XML that goes into a request. See next code section for more details:
Private Function makeSoapRequest(ByVal soapBody As String) As String
Dim req As WebRequest = WebRequest.Create(_soap_url)
Dim xml As String
xml = getSoapHeader() & soapBody
Dim buffer() As Byte = System.Text.Encoding.UTF8.GetBytes(xml)
req.ContentType = "text/xml; charset=utf-8"
req.Method = "POST"
req.Headers.Add("SOAPAction", "urn:Mage_Api_Model_Server_HandlerAction")
req.ContentLength = buffer.Length
Dim st As System.IO.Stream = req.GetRequestStream
st.Write(buffer, 0, buffer.Length)
st.Close()
Dim response As WebResponse
Try
response = req.GetResponse
Catch ex As WebException
response = ex.Response
End Try
st = response.GetResponseStream()
Dim reader As New StreamReader(st)
Dim responseFromServer As String = reader.ReadToEnd()
makeSoapRequest = responseFromServer
response.Close()
st.Close()
End Function
Below is the getSoapHeader() function. As noted the ns2 portion is only needed if you are using type="ns2:Map" which is what I needed for associative arrays:
Private Function getSoapHeader() As String
'ns2 is not always needed
getSoapHeader = "<?xml version=""1.0"" encoding=""UTF-8""?><SOAP-ENV:Envelope xmlns:SOAP-ENV=""http://schemas.xmlsoap.org/soap/envelope/"" xmlns:ns1=""urn:Magento"" xmlns:xsd=""http://www.w3.org/2001/XMLSchema"" xmlns:xsi=""http://www.w3.org/2001/XMLSchema-instance"" xmlns:SOAP-ENC=""http://schemas.xmlsoap.org/soap/encoding/"" xmlns:ns2=""http://xml.apache.org/xml-soap"" SOAP-ENV:encodingStyle=""http://schemas.xmlsoap.org/soap/encoding/""> " & vbCrLf
End Function
I am new to Web Crawling, and I am using HttpWebRequest to crawl data from sites.
As of now I was successfully able to crawl and get data from my wordpress site. This data was a simple user profile data. (like name, email, AIM id etc...)
Now as an exercise I want to crawl wikipedia, where I will search using the value entered into textbox at my end and then crawl wikipedia with the search value and get the appropriate title(s) from the search.
Now I have the following doubts/difficulties.
Firstly, is this even possible ? I have heard that wiki has robot.txt setup to block this. Though I have heard this only from a friend and hence not sure.
I am using the same procedure I used earlier, but I am not getting the required results.
Thanks !
Update :
After some explanation and help from #svick, I tried the below code, but still not able to get any value (see last line of code, there I am expecting an html markup of the search result page)
string searchUrl = "http://en.wikipedia.org/w/index.php?search=Wikipedia&title=Special%3ASearch";
var postData = new StringBuilder();
postData.Append("search=" + model.Query);
postData.Append("&");
postData.Append("title" + "Special:Search");
byte[] data2 = Crawler.GetEncodedData(postData.ToString());
var webRequest = (HttpWebRequest)WebRequest.Create(searchUrl);
webRequest.Method = "POST";
webRequest.UserAgent = "Crawling HW (http://yassershaikh.com/contact-me/)";
webRequest.AllowAutoRedirect = false;
ServicePointManager.Expect100Continue = false;
Stream requestStream = webRequest.GetRequestStream();
requestStream.Write(data2, 0, data2.Length);
requestStream.Close();
var responseCsv = (HttpWebResponse)webRequest.GetResponse();
Stream response = responseCsv.GetResponseStream();
// Todo Parsing
var streamReader = new StreamReader(response);
string val = streamReader.ReadToEnd();
// val is empty !! <-- this is my problem !
and here is my GetEncodedData method defination.
public static byte[] GetEncodedData(string postData)
{
var encoding = new ASCIIEncoding();
byte[] data = encoding.GetBytes(postData);
return data;
}
Pls help me on this.
You probably don't need to use HttpWebRequest. Using WebClient (or HttpClient if you're on .Net 4.5) will be much easier for you.
robots.txt doesn't actually block anything. If something doesn't support it (and .Net doesn't support it), it can access anything.
Wikipedia does block requests that don't have their User-Agent header set. And you should use an informative User-Agent string with your contact information.
A better way to access Wikipedia is to use its API, rather than scraping. This way, you will get an answer that's specifically meant to be read by a custom applications, formatted as XML or JSON. There are also dumps containing all information from Wikipedia available for download.
EDIT: The problem with your newly posted code is that your query returns a 302 Moved Temporarily response to the searched article, if it exists. Either remove the line that forbids AllowAutoRedirect, or add &fulltext=Search to your query, which will mean you won't get redirected.
I am having a RESTful service with the following method:
[WebInvoke]
string GetDataFromStringAsString(string xmlString);
My client call to the method is as below:
var client = new RestClient();
client.BaseUrl = serviceBaseUrl;
var request = new RestRequest(method){RequestFormat = DataFormat.Xml};
request.Resource = resourceUrl;
request.AddParameter("text/xml", requestBody,
ParameterType.RequestBody);
var response = client.Execute(request);
Let us take a string to post as "Hello World".
Now the string that i post to the above method gives me a 400 Bad
request. In order to get it working i had to wrap the above string in
a element as shown below:
<string xmlns="http://schemas.microsoft.com/2003/10/
Serialization/">Hello World</string>
Now when i post the above string i get a success response back from
the server.
Why is that i have to manually wrap the string to make it work. Is
there a way that i can achieve to post a string without doing the
above manually.
The only other way that I am aware of is to use stream as your input parameter. e.g.
[WebInvoke]
string GetDataFromStringAsString(stream xmlString);
The problem with .Net 4 WCF REST is that fundamentally WCF only knows how to pass two types of info, either XML or a stream of bytes. Personally, I would use WCF Web API instead of the standard WCF REST library because you are going run into lots more of these kinds of issues.