I am very new to skrape.it and try to do a simple request with it.
When using the HttpFetcher, I get the following error as soon as the response block is present:
Exception in thread "main" java.lang.NoClassDefFoundError: io/ktor/client/features/HttpTimeout
The function I wrote looks like this:
fun getDocumentByUrl(urlToScrape: String) = skrape(HttpFetcher) {
request {
url = urlToScrape
timeout = 10000
}
response {
htmlDocument {
div(".title_table_routenliste") { findAll {}}
}
}
}
Without the response block, the request itself works fine.
In my dependencies I added these two lines:
implementation("io.ktor:ktor-client-core:$ktor_version")
implementation("it.skrape:skrapeit:1.2.2")
What am I doing wrong?
Related
I have a Xamarin app that talks to an API. There is a certain scenario that happens when talking to the API in that a 401 (Unauthorized) exception is returned. This 401 (Unauthorized) is returned on purpose when the user account is made inactive so that even though the users token is still valid on the app they wouldn't be able to get any data back from the API.
I want to be able log the user out of the app, only when a 401 (Unauthorized) exception is thrown.
My API call looks like this:
public async Task<T> GetAsync<T>(string url)
{
_client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", _authToken?.AccessToken ?? this.GetToken().AccessToken);
var json = await _client.GetStringAsync(url);
return json.Deserialize<T>();
}
When the debugger reaches the var json = await _client.GetStringAsync(url); line a 401 (Unauthorized) exception is correctly thrown.
I want to be able to handle this 401 (Unauthorized) exception and log the user out of the app (preferably with an alert informing them of this).
I'm currently debugging on an Android device so I tried adding the following code to the MainActivity class.
protected override async void OnCreate(Bundle bundle)
{
AndroidEnvironment.UnhandledExceptionRaiser += AndroidEnvironmentOnUnhandledException;
}
private void AndroidEnvironmentOnUnhandledException(object sender, RaiseThrowableEventArgs e)
{
if(e.Exception.InnerException.GetBaseException().Message == "401 (Unauthorized)")
{
}
}
When the error is thrown I check if its a 401 (Unauthorized). It was here that I thought I would then log the user out of the app but I don't think this is the right direction.
Is there a best practice for handing this type of scenario that I am not aware of yet?
You could try to use try catch to warp var json = await _client.GetStringAsync(url) like the following code.
try
{
var json = await _client.GetStringAsync(url)
}
catch (WebException e)
{
using (WebResponse response = e.Response)
{
HttpWebResponse httpResponse = (HttpWebResponse)response;
Console.WriteLine("Error code: {0}", httpResponse.StatusCode);
using (Stream data = response.GetResponseStream())
using (var reader = new StreamReader(data))
{
// text is the response body
string text = reader.ReadToEnd();
if (text == "401 (Unauthorized)")
{
}
}
}
}
I'm getting a BadRequestException when using Jersey 2 and I'd like to get the URL that was used in the request and print it in the logs when the exception is caught.
Is there a property/method combination on the BadRequestException object that will return the URL? I don't see it in the JavaDocs, but it could have a name unrelated to "URL".
You can't get URI from BadRequestException. But you can get it from WebTarget you invoked the request on:
WebTarget target = ClientBuilder.newClient().target("http://localhost");
try {
String result = target.request().get(String.class);
} catch (BadRequestException bre) {
// get the URI
target.getUri();
}
If you don't want to use try-catch block:
Response response = target.request().get();
if (response.getStatus() == 200) {
// OK
} else {
// Handle other state.
}
I am implementing an API call to the MailChimp API in my web application. But this question is valid for any call to an API which accepts a simple HTTP GET.
I would like to catch any errors and allow my application to continue without issues even if the API call failed, The mailchimp servers and down, slow or unavailable.
MailChimp API Returns a serialized JSON Object. This is my code.
def listSubscribe = { apiurl, apikey, listid, email ->
def url = "${apiurl}?method=listSubscribe&apikey=${apikey}&id=${listid}&email_address=${email}"
}
try {
def url = new URL(listSubscribe(apiUrl,apiKey,listId,email))
return (url.text == 'true') ? true : false
} catch (MalformedURLException e) {
return false
} catch (java.net.UnknownHostException e) {
return false
}
Should I perform any other try/catch? How can I improve my code to make it safer for inaccesible API calls?
Solution
In order to make the call asynchronous and since I am using this code inside a Grails application I created a Quartz Job to execute the service containing the API Call.
class MailChimpListSubscribeJob {
def mailChimpService
def execute(context) {
mailChimpService.listSubscribe(context.mergedJobDataMap.get('email'))
}
}
The Service now uses a timeout and catches the generic Exception:
class MailChimpService {
def grailsApplication
def listSubscribe(email_address) {
def apiurl = grailsApplication.config.mailchimp.apiUrl
def apikey = grailsApplication.config.mailchimp.apiKey
def listid = grailsApplication.config.mailchimp.listId
listSubscribe(apiurl, apikey, listid, email_address)
}
def listSubscribe(apiurl, apikey, listid, email) {
try {
def cmdurl = "${apiurl}?method=listSubscribe&apikey=${apikey}&id=${listid}&email_address=${email}"
def url = new URL(cmdurl)
def response = url.getText(connectTimeout: 4 * 1000, readTimeout: 4 * 1000)
return (response == 'true') ? true : false
} catch (MalformedURLException e) {
return false
} catch (java.net.UnknownHostException e) {
return false
} catch (Exception e) {
return false
}
}
}
And inside my controllers:
MailChimpListSubscribeJob.triggerNow([email: 'myemail#example.com'])
Sergio - You may also want to do the following:
Enable timeout on this call in case you don't get any response from their system in a few seconds. This will take care of the events when their system is slow to respond, or offline.
If it's not a critical call that determines whether you can proceed with the rest of the code execution, then you should also consider making it asynchronous.
Catch the generic Exception as well in case you want to handle any unforeseen runtime exceptions that may occur.
When an Action is called and throws a specific exception, I use an ExceptionFilterAttribute that translate the error into a different response as HttpStatusCode.BadRequest. This has been working locally, but we pushed it to a server and now when I get the BadRequest I do not get any information in the reponse. What am I missing?
public override void OnException(HttpActionExecutedContext actionExecutedContext)
{
MyException ex = actionExecutedContext.Exception as MyException;
if (ex == null)
{
base.OnException(actionExecutedContext);
return;
}
IEnumerable<InfoItem> items = ex.Items.Select(i => new InfoItem
{
Property = i.PropertyName,
Message = i.ToString()
});
actionExecutedContext.Result = new HttpResponseMessage<IEnumerable<InfoItem>>(items, HttpStatusCode.BadRequest);
}
Edit: When I hit the service locally the body is included. It seems the problem is when hitting the service from a remote machine.
Try this:
GlobalConfiguration.Configuration.IncludeErrorDetailPolicy =
IncludeErrorDetailPolicy.Always
I am having trouble returning the correct HTTP error code for a "not found" in my WCF Web API code. Here is my api method ...
[WebInvoke(Method = "GET", UriTemplate = "{id}")]
[RequireAuthorisation]
public Customer GetCustomer(int id)
{
var customer = Repository.Find(id);
if (customer == null)
{
throw new HttpResponseException(HttpStatusCode.NotFound);
}
return customer;
}
I also have a logging handler ...
protected override bool OnTryProvideResponse(Exception exception, ref HttpResponseMessage message)
{
if (exception != null)
{
var msg = "Request failed.";
_logger.Error(exception, msg);
}
message = new HttpResponseMessage
{
StatusCode = HttpStatusCode.InternalServerError
};
return true;
}
What is happening is I am getting the following exception ...
HttpResponseException
"The response message returned by the Response property of this exception should be immediately returned to the client. No further handling of the request message is required."
... which my logging handler picks up and changes the response status code to a 500.
So, based on reading a few blog posts and answers on SO, I changed to this ...
if (customer == null)
{
WebOperationContext.Current.OutgoingResponse.SetStatusAsNotFound();
return null;
}
... but this now give me a 200. Which is clearly wrong.
So, what is the right way to do this? It seems as if the throwing of the HttpResponseException doesn't work and the code after gets executed.
The code snippet for your error handler is always changing the response message to 500 no matter what as you are explicitly setting the status always to 500.
It sounds like what you are trying to do is return a 500 ONLY if it is an application error. If that is the case you should check if the error exception is an HttpResponseException and just return rather than overriding.
As to WebOperationContext, don't use it at all with Web Api as it is basically no-op.
Hope this helps
Glenn