im trying to convert the call to the FORVO api (forvo.com) from a hardcoded url string to assigning the request parameters one by one to a ForvoRequestClass's properties that i created in C# via .AddParameter method...but when the request is made...it turns it into querystring format like this:
http://apifree.forvo.com/?key=[mykeygoeshere]&language=en&format=json&limit=1&order=rate-desc&sex=f&type=word&word=APPLE}
instead of like this:
http://apifree.forvo.com/key/[mykeygoeshere]/language/en/format=json/limit/1/order/rate-desc/sex/f/type/word/word/APPLE
i have tried various ways to call it using RestSharp...to no avail...any ideas...??
Ok...i believe i found the answer. On the github repo for RestSharp ( [1]: https://github.com/restsharp/RestSharp/wiki/ParameterTypes-for-RestRequest) I found the following helpful section:
UrlSegment
Unlike GetOrPost, this ParameterType replaces placeholder values in
the RequestUrl:
var rq = new RestRequest("health/{entity}/status");
rq.AddParameter("entity", "s2", ParameterType.UrlSegment);
When the request executes, RestSharp will try to match any
{placeholder} with a parameter of that name (without the {}) and
replace it with the value.
So the above code results in “health/s2/status” being the Url.
So with this info, I constructed my url string with placeholders for each of the parameters for Forvo like this:
/key/{key}/format/{format}/action/{action}/language/{language}/sex/{sex}/order/{order}/limit/{limit}/type/{type}/word/{word}
My final code to then do the substitution was this:
IRestClient client = new RestClient(soundRequest.SiteUrlValue);
request.AddParameter(soundRequest.ApiKeyName, soundRequest.ApiKeyValue, ParameterType.UrlSegment);
request.AddParameter(soundRequest.ActionName, soundRequest.ActionValue, ParameterType.UrlSegment);
request.AddParameter(soundRequest.LanguageName, soundRequest.LanguageValue, ParameterType.UrlSegment);
request.AddParameter(soundRequest.FormatName, soundRequest.FormatValue, ParameterType.UrlSegment);
request.AddParameter(soundRequest.SexName, soundRequest.SexValue, ParameterType.UrlSegment);
request.AddParameter(soundRequest.OrderName, soundRequest.OrderValue, ParameterType.UrlSegment);
request.AddParameter(soundRequest.LimitName, soundRequest.LimitValue, ParameterType.UrlSegment);
request.AddParameter(soundRequest.TypeName, soundRequest.TypeValue, ParameterType.UrlSegment);
request.AddParameter(soundRequest.WordName, soundRequest.WordValue, ParameterType.UrlSegment);
Related
return webClient//
.post()//
.uri(whatever.com)
.header("Authorization", "Bearer " + authToken)//
.header("userId", CLIENT_ID)//
.header("clientRequestId", requestId)//
.bodyValue(bodyValue())//
.retrieve()//
.bodyToMono(Responseclass.class)//
.block();
The above is working. But let's say I'm debugging and I just want to dump the raw response json into a String. How would I do that? toString() after retrieve doesn't work, and bodyToMono(String.class) didn't seem to work either. Either way it just printed the default toString value of the address of the pointer.
With Spring Webflux WebClient you can get response as String like this:
WebClient client = WebClient.create("http://someurl.de/something");
String responseBody = client.get().retrieve().toEntity(String.class)
.block().getBody();
In my German blog I wrote an article about using WebClient, there you will find more details around the code snippet above:
https://agile-coding.blogspot.com/2021/01/reactive-webclient.html
I have executed RestRequest and got non-typed IRestResponse.
How can I convert it to IRestResponse<T>?
For example IRestResponse<MyErrorData> or IRestResponse<MyData>?
You need to use the generic overload of Execute:
var client = new RestClient();
client.BaseUrl = BaseUrl;
request.AddParameter("AccountSid", _accountSid, ParameterType.UrlSegment);
var response = client.Execute<T>(request);
Execute<T> is the key to getting back a typed response.
I found that, depending on the version of RestSharp you have installed, Execute<T> threw a compiler error. (Tho it seems fine in v106.15.)
Another option is to cast it, which seems to work when the first option doesn't:
RestClient client = new RestClient();
IRestResponse<T> response = (IRestResponse<T>)client.Execute(request);
Also, don't forget that your method (or class) must be decorated with the T type parameter:
Eg
partial void InterceptResponse<T>(IRestRequest request, ref IRestResponse<T> response)
...
RestClient client = new RestClient();
IRestResponse<T> response = client.Execute<T>(request);
(or IRestResponse<T> response = (IRestResponse<T>)client.Execute(request);)
....
(In this example, I'm intercepting a RestResponse, doing something, such as re-calling, and passing back the new response as ref response.)
USing Wicket 6.17 and servlet 2.5, I have a form that allows file upload, and also has ReCaptcha (using Recaptcha4j). When the form has ReCaptcha without file upload, it works properly using the code:
final HttpServletRequest servletRequest = (HttpServletRequest ) ((WebRequest) getRequest()).getContainerRequest();
final String remoteAddress = servletRequest.getRemoteAddr();
final String challengeField = servletRequest.getParameter("recaptcha_challenge_field");
final String responseField = servletRequest.getParameter("recaptcha_response_field");
to get the challenge and response fields so that they can be validated.
This doesn't work when the form has the file upload because the form must be multipart for the upload to work, and so when I try to get the parameters in that fashion, it fails.
I have pursued trying to get the parameters differently using ServletFileUpload:
ServletFileUpload fileUpload = new ServletFileUpload(new DiskFileItemFactory(new FileCleaner()) );
String response = IOUtils.toString(servletRequest.getInputStream());
and
ServletFileUpload fileUpload = new ServletFileUpload(new DiskFileItemFactory(new FileCleaner()) );
List<FileItem> requests = fileUpload.parseRequest(servletRequest);
both of which always return empty.
Using Chrome's network console, I see the values that I'm looking for in the Request Payload, so I know that they are there somewhere.
Any advice on why the requests are coming back empty and how to find them would be greatly appreciated.
Update: I have also tried making the ReCaptcha component multipart and left out the file upload. The result is still the same that the response is empty, leaving me with the original conclusion about multipart form submission being the problem.
Thanks to the Wicket In Action book, I have found the solution:
MultipartServletWebRequest multiPartRequest = webRequest.newMultipartWebRequest(getMaxSize(), "ignored");
// multiPartRequest.parseFileParts(); // this is needed since Wicket 6.19.0+
IRequestParameters params = multiPartRequest.getRequestParameters();
allows me to read the values now using the getParameterValue() method.
I am new to Web Crawling, and I am using HttpWebRequest to crawl data from sites.
As of now I was successfully able to crawl and get data from my wordpress site. This data was a simple user profile data. (like name, email, AIM id etc...)
Now as an exercise I want to crawl wikipedia, where I will search using the value entered into textbox at my end and then crawl wikipedia with the search value and get the appropriate title(s) from the search.
Now I have the following doubts/difficulties.
Firstly, is this even possible ? I have heard that wiki has robot.txt setup to block this. Though I have heard this only from a friend and hence not sure.
I am using the same procedure I used earlier, but I am not getting the required results.
Thanks !
Update :
After some explanation and help from #svick, I tried the below code, but still not able to get any value (see last line of code, there I am expecting an html markup of the search result page)
string searchUrl = "http://en.wikipedia.org/w/index.php?search=Wikipedia&title=Special%3ASearch";
var postData = new StringBuilder();
postData.Append("search=" + model.Query);
postData.Append("&");
postData.Append("title" + "Special:Search");
byte[] data2 = Crawler.GetEncodedData(postData.ToString());
var webRequest = (HttpWebRequest)WebRequest.Create(searchUrl);
webRequest.Method = "POST";
webRequest.UserAgent = "Crawling HW (http://yassershaikh.com/contact-me/)";
webRequest.AllowAutoRedirect = false;
ServicePointManager.Expect100Continue = false;
Stream requestStream = webRequest.GetRequestStream();
requestStream.Write(data2, 0, data2.Length);
requestStream.Close();
var responseCsv = (HttpWebResponse)webRequest.GetResponse();
Stream response = responseCsv.GetResponseStream();
// Todo Parsing
var streamReader = new StreamReader(response);
string val = streamReader.ReadToEnd();
// val is empty !! <-- this is my problem !
and here is my GetEncodedData method defination.
public static byte[] GetEncodedData(string postData)
{
var encoding = new ASCIIEncoding();
byte[] data = encoding.GetBytes(postData);
return data;
}
Pls help me on this.
You probably don't need to use HttpWebRequest. Using WebClient (or HttpClient if you're on .Net 4.5) will be much easier for you.
robots.txt doesn't actually block anything. If something doesn't support it (and .Net doesn't support it), it can access anything.
Wikipedia does block requests that don't have their User-Agent header set. And you should use an informative User-Agent string with your contact information.
A better way to access Wikipedia is to use its API, rather than scraping. This way, you will get an answer that's specifically meant to be read by a custom applications, formatted as XML or JSON. There are also dumps containing all information from Wikipedia available for download.
EDIT: The problem with your newly posted code is that your query returns a 302 Moved Temporarily response to the searched article, if it exists. Either remove the line that forbids AllowAutoRedirect, or add &fulltext=Search to your query, which will mean you won't get redirected.
I am having a RESTful service with the following method:
[WebInvoke]
string GetDataFromStringAsString(string xmlString);
My client call to the method is as below:
var client = new RestClient();
client.BaseUrl = serviceBaseUrl;
var request = new RestRequest(method){RequestFormat = DataFormat.Xml};
request.Resource = resourceUrl;
request.AddParameter("text/xml", requestBody,
ParameterType.RequestBody);
var response = client.Execute(request);
Let us take a string to post as "Hello World".
Now the string that i post to the above method gives me a 400 Bad
request. In order to get it working i had to wrap the above string in
a element as shown below:
<string xmlns="http://schemas.microsoft.com/2003/10/
Serialization/">Hello World</string>
Now when i post the above string i get a success response back from
the server.
Why is that i have to manually wrap the string to make it work. Is
there a way that i can achieve to post a string without doing the
above manually.
The only other way that I am aware of is to use stream as your input parameter. e.g.
[WebInvoke]
string GetDataFromStringAsString(stream xmlString);
The problem with .Net 4 WCF REST is that fundamentally WCF only knows how to pass two types of info, either XML or a stream of bytes. Personally, I would use WCF Web API instead of the standard WCF REST library because you are going run into lots more of these kinds of issues.