webclient().downloadstring won't load everything - vb.net

Hello I am using visual basic class webclient to download an html document into a string.
Here is the code
Dim htmlsource As String
htmlsource = New System.Net.WebClient().DownloadString("website")
RichTextBox1.Text = htmlsource
The problem is that (webclient().downloadstring) only downloads a part of the website loads as the rest of the html document loads a few seconds after the webpage is loaded. Is there a work around this? Like putting a delay or similar

try this
WebClient Client = new WebClient();
string tempHtml = Client.Encoding.GetString(Client.DownloadData(IncomingAddress));
dim Client as new WebClient()
dim tempHtml as String = Client.Encoding.GetString(Client.DownloadData(IncomingAddress))
your problem is very familiar with all internet spiders, your content come from ajax, you cannot see it because it load in a different request, for doing this is you need to work through browser or webkit...

Related

Jsoup only fetches few lines of the HTML from start which is not even 25%

When i try to do CTRL+U on website then also its more then what i get from jsoup. The site am using is Open SAP -> https://open.sap.com/courses
Have tried timeout and maxbodysize along with jsoup.connect.
Right now my code looks like this:
private static String getHtml(String location) throws IOException {
URL url = new URL(location);
URLConnection conn = url.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String input;
StringBuilder builder = new StringBuilder();
while((input = in.readLine()) != null)
{
builder.append(input);
}
return builder.toString();
}
document = Jsoup.parse(getHtml(URL));
But still same HTML returned. By selenium its possible but it a bit slow so any other way to achieve this?
Because by aim is to find out the links of the courses and then load them to find their course summary which with selenium will be too slow.
Please suggest what can be done here.
The page content of this page is constructed inside your browser based on js. You need a framework with js support to do this.
Using HtmlUnit i got the page like this
String url = "https://open.sap.com/courses";
try (final WebClient webClient = new WebClient(BrowserVersion.FIREFOX_68)) {
webClient.getOptions().setThrowExceptionOnScriptError(false);
HtmlPage page = webClient.getPage(url);
webClient.waitForBackgroundJavaScriptStartingBefore(10_000);
System.out.println("-------------------------------");
System.out.println(page.asText());
System.out.println("-------------------------------");
}
HtmlUnit has a rich API to do everything you like with the page object like searching for controls/content, clicking controls or extracting the text from parts of the page.

Uploading an image to a site via their API

I have a bot running in an online game which the staff use to draw out the map's levels. It uploads these pics to Imgur using its API, but Imgur isn't always up and running so I'd like to add a 2nd host as a backup plan.
I discovered this site which looks like a fine choice: https://boring.host/api
To get an API key, sign up (fast, no confirmation needed) then go to "My Account" by clicking your name on the top right.
Looking at the API info, it looks like I just need to send a HTTP POST to "https://api.boring.host/api/v1/upload" and include the "api_token" and "file" parameters.
When I upload to Imgur, I write the image to a MemoryStream then convert that to a Base64String:
Convert.ToBase64String(bytImage)
So I tried the same thing for this:
Public Function UploadBoringHost(Dim bm as Bitmap) As String
'Write image to memorystream
Dim bytImage As Byte()
Using stream As New System.IO.MemoryStream
bm.Save(stream, System.Drawing.Imaging.ImageFormat.Png)
bytImage = stream.ToArray
End Using
'Upload
Using client As New WebClient
Dim reqparm As New Specialized.NameValueCollection
reqparm.Add("api_token", strApiKey)
reqparm.Add("file", Convert.ToBase64String(bytImage))
Dim responsebytes = client.UploadValues("https://api.boring.host/api/v1/upload", "POST", reqparm)
Dim responsebody = (New Text.UTF8Encoding).GetString(responsebytes)
End Using
End Function
Trying it this way, I get the following web exception: "The remote server returned an error: (422) Unprocessable Entity"
My only guess is that it's expecting the image data to be passed through in some other format. Any ideas?

How to get the loading time of a web page in Windows Service Application using HttpWebRequest

I'm looking for code that will help me get the loading time of a web page without Using WebBrowser() in a Windows Service Application.
I run through different methods, but I don't quite get it.
Please help me solve this problem.
This function should do the trick:
Public Function WebpageResponseTime(ByVal URL As String) As TimeSpan
Dim sw As New System.Diagnostics.Stopwatch
sw.Start()
Dim wRequest As WebRequest = HttpWebRequest.Create(URL)
Using httpResponse As HttpWebResponse = DirectCast(wRequest.GetResponse(), HttpWebResponse)
If httpResponse.StatusCode = HttpStatusCode.OK Then
sw.Stop()
Return sw.Elapsed
End If
End Using
End Function
This will only take the downloading of the source code into account. If you want to calculate how long time it takes to download the source code AND render the page you'd have to use the WebBrowser class.
How it works:
The function declares and starts a Stopwatch which will be used for calculating how long the operation took, then it creates a web request to the specified URL. It downloads the entire page's source code (via the HttpWebResponse) and after that checks the response's StatusCode.
StatusCode.OK (which is HTTP status code 200) means that the request succeeded and that the requested information (the web page's source code) is in the response, but as we're not gonna use the source code for anything we let the response get disposed later by the Using/End Using block.
And lastly the function stops the Stopwatch and returns the elapsed time (how long it took to download the web page's source) to you.
Example use:
Dim PageLoadTime As TimeSpan = WebpageResponseTime("http://www.microsoft.com/")
MessageBox.Show("Response took: " & PageLoadTime.ToString())

Can somebody give me an example of getting the content of a page from a URL in vb.net for windows 8?

I am very new to vb/.net and I'm trying to do something that I can do easily in classic vb. I want to get the source html for a webpage from the URL.
I'm using vb.net in Visual Studio Express for Windows 8.
I've read loads of stuff that talk about HttpWebRequest, but I can't get it to work properly.
I did at one point have it returning the html header, but I want to content of the page. Now, I can't even get it back to giving me the header. Ultimately, I want to process the html returned which I'll do (to begin with) the old-fashioned way and process the returned html as a string, but for now I'd like to just get the page.
The code I've got is:
Dim URL As String = "http://www.crayola.com/"
Dim request As System.Net.HttpWebRequest = System.Net.HttpWebRequest.Create(New Uri(URL))
txtHTML.Text = request.GetRequestStreamAsync().ToString()
Can anyone help me with an example to get me going please?
You're trying to use an Async method in a synchronous way, which won't make any sense. If you're using .NET 4.5, you can try marking the calling method with Async and then using the Await keyword when calling GetRequestStreamAsync.
Public Sub MyDownloaderMethod()
Dim URL As String = "http://www.crayola.com/"
Dim request As System.Net.HttpWebRequest
= System.Net.HttpWebRequest.Create(New Uri(URL))
' Use the Await keyword wait for the async task to complete.
Dim response = request.GetResponseAsync()
txtHTML.Text = response.GetResponseStream().ToString()
End Function
See the following MSDN article for more information on async programming with the Await keyword: http://msdn.microsoft.com/en-us/library/vstudio/hh191443.aspx
Edit
You are receiving your error because you're trying to get the Request stream (what you send the server), and what you really want is the Response stream (what the server sends back to you). I've updated my code to get the WebResponse from your WebRequest and then retrieve the stream from that.
Public Shared Function GetWebPageString(ByVal address As Uri) As String
Using client As New Net.WebClient()
Return client.DownloadString(address)
End Using
End Function
There is also DownloadStringAsync if you don't want to block
request.GetRequestStreamAsync() is probably not a method. I think you're cribbing code from a site where someone wrote their own add-on methods to HttpWebRequest. Try request.GetResponse() to return a response object, then in the response object you can inspect the stream and convert it to text if you need to.
This worked for me in VB.Net 4.5
Public Async Sub GetHTML()
Dim PageHTML as string
Dim client As New HttpClient
Dim getStringTask As Task(Of String) = client.GetStringAsync(PageURL)
PageHTML = Await getStringTask
MsgBox(PageHTML)
End Sub

Windows Phone VB Web Request

does anybody has an idea how I can perform an asynchronus Post Request in VB.Net for Windows Phone 8?
I tried a lot but nothing worked... also this http://msdn.microsoft.com/de-de/library/system.net.httpwebrequest.begingetrequeststream.aspx didn't work.
Thanks a lot.
I had to figure this out for myself a while ago. Let me see what I can do to help.
Posting a web request is actually simpler than that link shows. Here's what I do.
First, I create a MultipartFormDataContent:
Dim form as New MultipartFormDataContent()
Next, I add each string I want to send like this:
form.Add(New StringContent("String to sent"), "name of the string you are sending")
Next, create a HttpClient:
Dim httpClient as HttpClient = new HttpClient()
Next, we'll create a HttpResponseMessage and post your information to the url of your choice:
Dim response as HttpResponseMessage = Await httpClient.PostAsync("www.yoururl.com/wherever", form)
Then, I usually need the response as a string, so I read the response to a string:
Dim responseString as String = Await response.Content.ReadAsStringAsync()
This will give you the response you wanted, if that's what you wanted.
Here's an example of a method I use:
Public Async Function GetItems() As Task
Dim getUrl As String = "https://myapiurl.com/v3/get"
Dim responseText As String = String.Empty
Dim detailType As String = "complete"
Try
Dim httpClient As HttpClient = New HttpClient()
Dim form As New MultipartFormDataContent()
form.Add(New StringContent(roamingSettings.Values("ConsumerKey").ToString()), "consumer_key")
form.Add(New StringContent(roamingSettings.Values("access_token").ToString()), "access_token")
form.Add(New StringContent(detailType.ToString()), "detailType")
Dim response As HttpResponseMessage = Await httpClient.PostAsync(getUrl, form)
responseText = Await response.Content.ReadAsStringAsync()
Catch ex As Exception
End Try
End Function
If you aren't using the Http client libraries, you need to install them like this:
What you need to do to use the HttpClient, is to navigate in Visual Studio, go to Tools->Library Package Manager->Manage Nuget Packages for this solution. When there, search the online section for HttpClient and make sure you have "Include Prerelease" selected in the listbox above the results. (Default is set to "Stable Only")
Then install the package with the ID of Microsoft.Net.Http
Then you'll need to add an Import statement at the beginning of the document you are using it in.
Let me know if this is what you were looking for.
Thanks,
SonofNun