Getting the number of google search hits with google api - api

I am trying to get the number of search results with a custum search engine. I tried the following code:
String charset = "UTF-8";
String google = "https://www.googleapis.com/customsearch/v1?key={mykey}" +
"&cx={mycxcode}&q=" +URLEncoder.encode(searchString, charset) +
")&fields=queries(request(totalResults)";
String userAgent = "Mozilla/5.0 (Windows; U; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)";
String totalResultsElementText = Jsoup.connect(google).userAgent(userAgent).ignoreContentType(true).get().text();
For this code i get the following error message:
org.jsoup.HttpStatusException: HTTP error fetching URL. Status=400, URL={myurl}
What am i doing wrong?

The url you are creating has error. You should change parameter fields to
&fields=queries/request(totalResults), because request is a field of queries, but not an array.
More about partial selectors can be found here.
String charset = "UTF-8";
String google = "https://www.googleapis.com/customsearch/v1?key={mykey}" +
"&cx={mycxcode}&q=" +URLEncoder.encode(searchString, charset) +
"&fields=queries/request(totalResults)";
String userAgent = "Mozilla/5.0 (Windows; U; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)";
String totalResultsElementText = Jsoup.connect(google).userAgent(userAgent).ignoreContentType(true).get().text();
Your result is in form of JSON so you can extract final value like that:
JSONObject json = new JSONObject(totalResultsElementText);
System.out.println(json.getJSONObject("queries").getJSONArray("request").getJSONObject(0).get("totalResults"));

Related

System.Net.WebException: 'The remote server returned an error: (463).' [duplicate]

This question already has answers here:
How solve HTTP request failed! HTTP/1.1 463?
(2 answers)
Closed 3 years ago.
Im trying to make a tool that checks if a user exists but i get the error 463.
url im using (https://www.habbo.nl/habbo-imaging/avatarimage?hb=image&user=123)
Public Sub checkAccount()
Dim request As System.Net.HttpWebRequest = System.Net.HttpWebRequest.Create("https://www.habbo.nl/habbo-imaging/avatarimage?hb=image&user=" + userToCheck)
Dim repsonse As System.Net.HttpWebResponse = request.GetResponse()
Dim sReader As System.IO.StreamReader = New System.IO.StreamReader(repsonse.GetResponseStream)
Dim Habboresult As String = sReader.ReadToEnd()
If Habboresult.Contains("HTTP Status 404 – Not Found") Then
'add user to listbox of available names
freeName()
Else
'add user to listbox of names that are already in use
usedName()
End If
End Sub
Image of the error
Even I’m not agree with your approach (in checking for new available user names), in order to fix your code and doing it running you have to add this instruction .UserAgent after New declaration of "request" Object (like code below shows)
Dim request As System.Net.HttpWebRequest = CType(System.Net.HttpWebRequest.Create("https://www.habbo.nl/habbo-imaging/avatarimage?hb=image&user=123"), Net.HttpWebRequest)
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.246"
Staying in context (always based in you code) is wasted invoking all those classes for a downloaded string (are you shure you need to treat this as a string instead of bytes?? then if byte.length > 0…...).
Instead you can use three lines of code which are (for string data):
Dim client As Net.WebClient = New Net.WebClient()
client.Headers.Add("User-Agent" , "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.246")
Dim reply As String = client.DownloadString("https://www.habbo.nl/habbo-imaging/avatarimage?hb=image&user=123")
Or (for bytes data to convert in an image or testing it length)
Dim client As Net.WebClient = New Net.WebClient()
client.Headers.Add("User-Agent" , "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.246")
Dim imageBytes = client.DownloadData("https://www.habbo.nl/habbo-imaging/avatarimage?hb=image&user=123")
Edit: Check this link out https://stackoverflow.com/a/49956632/12808204 see if it helps.
I may be speaking from ignorance here, as networking isn't my forté, but the error range 452-499 isn't defined by an official RFC, and so what 463 means is likely implementation specific. Some cursory googling seems to support this, that this range is used for own defined error codes(But don't take my word as law on this). 4xx errors do generally refer to client errors though, i.e there may be an issue with your request. Maybe check that the string argument for System.Net.HttpWebRequest.Create() is correct? Break it out to its own variable, and make sure userToCheck is actually defined when the function checkAccount() is called.
Without more info about the site or API you're interfacing with, I don't have more to give. Provide some more background info?

Using WebClient to Download CSV File From SSRS Report

I'm trying to download a csv file from an SSRS report using the following code.
Const URI As String = "https://blah.blah.com/blah/_layouts/15/ReportServer/RSViewerPage.aspx?rv:RelativeReportUrl=/blah/Production%20Reports/The_File.rdl&rs:format=csv"
Const DESTINATION As String = "C:\MyFile.csv"
Using myWebClient As WebClient = New WebClient()
With myWebClient
.Headers.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")
.Headers.Add("Accept-Encoding", "gzip, deflate, sdch, br")
.Headers.Add("Accept-Language", "en-US,en;q=0.8")
.Headers.Add("Content-Disposition", "attachment; filename=%22The%5FFile.csv%22")
.Headers.Add("Content-Encoding", "gzip")
.Headers.Add("Content-Type", "text/csv")
.Headers.Add("Vary", "Accept-Encoding")
.Headers.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36")
.Headers.Add("Upgrade-Insecure-Requests", "1")
.Headers.Add("Referer", URI)
.Headers.Add("Cache-Control", "private")
.Credentials = New NetworkCredential("<my username>", "<my password>")
.DownloadFile(URI, DESTINATION)
End With
End Using
The problem is that the file that gets downloaded isn't a csv file. When I open it in any text editor, all I see are "garbage" characters which seem like some sort of encoding is going on. If I comment out the "Accept-Encoding" header and rerun the code, I get the code of the resulting HTML page - not the csv file I need. Anyone know how I can download the file correctly? BTW, I'm not sure all of the headers I added are necessary.
You need to change the URI constant from this:
Const URI As String = "https://blah.blah.com/blah/_layouts/15/ReportServer/RSViewerPage.aspx?rv:RelativeReportUrl=/blah/Production%20Reports/The_File.rdl&rs:format=csv"
To this:
Const URI As String = "https://blah.blah.com/blah/_layouts/15/ReportServer/RSViewerPage.aspx?rv:RelativeReportUrl=/blah/Production%20Reports/The_File.rdl&rs:format=csv&rs:Command=Render"
You are missing the Command=Render part of the Uri.

Custom User Agent for a WebView

can I set a custom User Agent for a WebView?
I need to show mobile style of websites.
It's easy to do:
string ua = "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X)" + "AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25";
var httpRequestMessage = new HttpRequestMessage(HttpMethod.Get, new Uri(url));
httpRequestMessage.Headers.Add("User-Agent",ua);
webView1.NavigateWithHttpRequestMessage(hrm);
Per this MSDN Forum posting you cannot. Could you host a lightweight proxy service (say Azure Web Site) to proxy the request for you?
You can load HTML with custom user agent and then pass the html to WebView
Loading html
var handler = new HttpClientHandler {AllowAutoRedirect = false};
var client = new HttpClient(handler);
client.DefaultRequestHeaders.Add("user-agent",
"Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2;
WOW64; Trident/6.0)");
var response = await client.GetAsync(url);
response.EnsureSuccessStatusCode();
var html = await response.Content.ReadAsStringAsync();
assign html to WebView
WebView.NavigateToString(html);

ASP.NET Web API Basic Authentication Authorisation Header

I have a BasicAuthenticationAttribute that inspects the Authorisation header in the request but despite it being present, it still believes the Authorisation header is null:
public class BasicAuthenticationAttribute : ActionFilterAttribute
{
public override void OnActionExecuting(HttpActionContext actionContext)
{
if (actionContext.Request.Headers.Authorization == null)
{
actionContext.Response = new HttpResponseMessage(HttpStatusCode.Unauthorized);
}
...
If I inspect actionContext.Request.Headers I can see Authorization listed:
{Connection: Keep-Alive
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-gb
Authorization: REDACTED_BUT_PRESENT==
Host: localhost:44300
Referer: https://localhost:44300/
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; InfoPath.3; .NET4.0E)
}
Update
I have just inspected the full request headers and they look like this... I can see an Authorization header in the first section, but the Authorization header in the second section is clearly null.
request.Headers
{Connection: Keep-Alive
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-gb
Authorization: REDACTED_BUT_PRESENT==
Host: localhost:1734
Referer: http://localhost:1734/
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; InfoPath.3; .NET4.0E)
}
base {System.Net.Http.Headers.HttpHeaders}: {Connection: Keep-Alive
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-gb
Authorization: VXNlcjpQYXNzd29yZA==
Host: localhost:1734
Referer: http://localhost:1734/
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; InfoPath.3; .NET4.0E)
}
Accept: {*/*}
AcceptCharset: {}
AcceptEncoding: {gzip, deflate}
AcceptLanguage: {en-gb}
Authorization: null
CacheControl: null
... removed for brevity ...
Warning: {}
If you get stuck on this, you can get the header using:
var header = request.Headers.FirstOrDefault(h => h.Key.Equals("Authorization"));
But not via
var header = request.Headers.Authorization;
I noticed myself that if the Authorization-header only contained the key/token, the request.Headers.Authorization wouldn't be initiated properly because it's looking for a scheme as well in the format <Scheme> <key/token>, i.e. Authorization: Token VXNlcjpQYXNzd29yZA==, then the Authorization wouldn't be null anymore and contain request.Headers.Authorization.Scheme = "Token" and request.Headers.Authorization.Parameter = "VXNlcjpQYXNzd29yZA=="
I've posted my own example of a Basic Authentication Attribute. Maybe this gives you some hints.
I use:
HttpContext.Current.Request.Headers["Authorization"];
And here is the link to the complete solution:
http://remy.supertext.ch/2012/04/basic-http-authorization-for-web-api-in-mvc-4-beta/
Though, this thread is very old but it might help others if I share how did I resolve it in my case:
Request should contain
Authorization: Basic VXNlcjpQYXNzd29yZA==
instead of:
Authorization: VXNlcjpQYXNzd29yZA==
so following change in request may solve the problem:
client.Headers.Add("Authorization", "Basic VXNlcjpQYXNzd29yZA==");
Adding more information to #finstas's answer.
Authorization is null because well defined HTTP headers like Accept, Authorization and many more are parsed when creating the HttpRequestHeaders class. Hence if the request comes in with a format different from what .NET accepts for that header then that specific property will be null.
Below is the decompiled code from the AuthenticationHeaderValue class responsible for parsing the Authorization header. Similarly there are other classes for the different HTTP headers which do the same.
Hope this sheds more info into why there needs to be a space between Token and the value.
internal static int GetAuthenticationLength(string input, int startIndex, out object parsedValue)
{
parsedValue = (object) null;
if (string.IsNullOrEmpty(input) || startIndex >= input.Length)
return 0;
int tokenLength = HttpRuleParser.GetTokenLength(input, startIndex);
if (tokenLength == 0)
return 0;
AuthenticationHeaderValue authenticationHeaderValue = new AuthenticationHeaderValue();
authenticationHeaderValue.scheme = input.Substring(startIndex, tokenLength);
int startIndex1 = startIndex + tokenLength;
int whitespaceLength = HttpRuleParser.GetWhitespaceLength(input, startIndex1);
int index = startIndex1 + whitespaceLength;
if (index == input.Length || (int) input[index] == 44)
{
parsedValue = (object) authenticationHeaderValue;
return index - startIndex;
}
if (whitespaceLength == 0)
return 0;
int startIndex2 = index;
int parameterEndIndex = index;
if (!AuthenticationHeaderValue.TrySkipFirstBlob(input, ref index, ref parameterEndIndex) || index < input.Length && !AuthenticationHeaderValue.TryGetParametersEndIndex(input, ref index, ref parameterEndIndex))
return 0;
authenticationHeaderValue.parameter = input.Substring(startIndex2, parameterEndIndex - startIndex2 + 1);
parsedValue = (object) authenticationHeaderValue;
return index - startIndex;
}

HttpWebRequest SSL Authorization form

I've never tried before, but now I really need to get through authorization on Sprint's site (www.sprint.com).
Could you guys help me to understand how this actually works?
I'm trying to do like this, but obviously I'm missing something. Either something about cookies
or ssl or other stuff, I don't know.
HttpWebRequest webRequest = (HttpWebRequest)HttpWebRequest.Create(
"https://sso.sprintpcs.com/sso/Login.do");
CookieContainer cookieContainer = new CookieContainer();
webRequest.CookieContainer = cookieContainer;
webRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0;
chromeframe; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729;
.NET CLR 3.0.30729; Tablet PC 2.0; .NET4.0C; .NET4.0E)";
webRequest.Accept = "image/jpeg, application/x-ms-application, image/gif, application/xaml+xml,
image/pjpeg, application/x-ms-xbap, application/x-shockwave-flash,
application/vnd.ms-excel, application/msword, */*";
webRequest.Method = "POST";
webRequest.Host = "manage.sprintpcs.com";
string strUserId = "kindauser";
string strPass = "kindapass";
ASCIIEncoding encoding = new ASCIIEncoding();
string postData = "userid=" + strUserId + "&password="
+ strPass + "&userExperince=USC allowlogin=false";
byte[] data = encoding.GetBytes(postData);
Stream requestStream = webRequest.GetRequestStream();
requestStream.Write(data,0,data.Length);
HttpWebResponse myHttpWebResponse = (HttpWebResponse)webRequest.GetResponse();
I would do the following - and this applies to all cases where you want to interact wit a website.
1) get firefox, along with firebug extension.
2) clear the firefox content and cookie cache
3) use firefox to do the scenario - logging into the website, for eg.
4) At this point, firebug shows you the exact sequence of requests sent along withh the cookie headers, etc.
5) Now try to replicate this using code.