I've created a script in vba using IE to keep clicking on the Load more hits button located at the bottom of a webpage until there is no such button is left.
Here is how my script can populate that button: In the site's landing page there is a dropdown named Type. The script can click on that Type to unfold the dropdown then it clicks on some corporate bond checkbox among the options. Finally, it clicks on the apply button to populate the data. However, that load more hits button can be visible at the bottom now.
My script can follow almost all the steps exactly what I described above. The only thing I am struggling to solve is that the script seems to get stuck after clicking on that button 3/4 times.
How can I rectify my script to keep clicking on that Load more hits button until there is no such button is left?
Website link
I've tried so far:
Sub ExhaustLoadMore()
Dim IE As New InternetExplorer, I As Long
Dim Html As HTMLDocument, post As Object, elem As Object
Dim CheckBox As Object, btnSelect As Object
With IE
.Visible = True
.navigate "https://www.boerse-stuttgart.de/en/tools/product-search/bonds"
While .Busy Or .readyState < 4: DoEvents: Wend
Set Html = .document
Do: Loop Until Html.querySelectorAll(".bsg-loader-ring__item").Length = 0
Html.querySelector("#bsg-filters-btn-bgs-filter-3").Click
Do: Set CheckBox = Html.querySelector("#bsg-checkbox-3053"): DoEvents: Loop While CheckBox Is Nothing
CheckBox.Click
Set btnSelect = Html.querySelector("#bsg-filters-menu-bgs-filter-3 .bsg-btn__label")
Do: Loop While btnSelect.innerText = "Close"
btnSelect.Click
Do: Loop Until Html.querySelectorAll(".bsg-loader-ring__item").Length = 0
Do: Set elem = Html.querySelector(".bsg-table__tr td"): DoEvents: Loop While elem Is Nothing
Do
Set post = Html.querySelector(".bsg-searchlist__load-more button.bsg-btn--juna")
If Not post Is Nothing Then
post.ScrollIntoView
post.Click
Application.Wait Now + TimeValue("00:00:05")
Else: Exit Do
End If
Loop
End With
End Sub
I've tried with selenium but that seems to be way slower. However, it keeps clicking on the load more button after a long wait in between even when no hardcoded wait within it. In case of selenium: I wish to have any solution which might help reduce it's execution time.
Sub ExhaustLoadMore()
Const Url$ = "https://www.boerse-stuttgart.de/en/tools/product-search/bonds"
Dim driver As New ChromeDriver, elem As Object, post As Object
With driver
.get Url
Do: Loop Until .FindElementsByCss(".bsg-loader-ring__item").count = 0
.FindElementByCss("#bsg-filters-btn-bgs-filter-3", timeOut:=10000).Click
.FindElementByXPath("//label[contains(.,'Corporate Bond')]", timeOut:=10000).Click
.FindElementByXPath("//*[#id='bsg-filters-menu-bgs-filter-3']//button", timeOut:=10000).Click
Do: Loop Until .FindElementsByCss(".bsg-loader-ring__item").count = 0
Set elem = .FindElementByCss(".bsg-table__tr td", timeOut:=10000)
Do
Set post = .FindElementByCss(".bsg-searchlist__load-more button.bsg-btn--juna", timeOut:=10000)
If Not post Is Nothing Then
post.ScrollIntoView
.ExecuteScript "arguments[0].click();", post
Do: Loop Until .FindElementsByCss("p.bsg-searchlist__info--load-more").count = 0
Else: Exit Do
End If
Loop
Stop
End With
End Sub
I have studied a bit your website, and since I could not say all of this into a single comment I have decided to post an answer (even though it doesn't provide with a concrete solution, but just with an "answer" and maybe some tips).
The answer to your question
How can I rectify my script to keep clicking on that Load more hits button until there is no such button is left?
Unfortunately, it's just not your fault. The website you are targeting is working through WebSocket communication between the web client (your browser) and the web server providing with the prices you are trying to scrape. You can see it as follows:
Imagine it like this:
When you first load your webpage, the web socket is initialized and the first request is sent (Web client: "Hey server, give me the first X results", Web server: "Sure, here you go").
Every time you click on the "Load more results" button, the Web client (important: re-using the same WS connection) keeps on asking for X new results to the web server.
So, the communication keeps on going on for some time. At some point, out of your control, it happens that the web socket just dies. It's enough to look at the JavaScript console while clicking on the "Load more results" button: you will see the request going through until at some point you don't just see a NullPointerException raised:
If you click on the last line of the stack before the exception, you will see that it's because of the web socket:
The error speaks clearly: cannot read .send() on null, meaning that _ws (the web socket) is gone.
Starting from now, you can forget about your website. When you click on the button "Load more results", the web client will ask the web socket to deliver the new request to the web server, but the web socket is gone so goodbye communications between the two, and so (unfortunately) goodbye the rest of your data.
You can verify this by just going a bit upper in the stack:
As you can see above, we have:
A message logged in the console saying "performSearch params ...") just before posting the new data request
The post of the new data request
A message logged in the console saying "performed search with result ...") just after posting the new data request
While the web socket is still alive, everytime you click on "Load more results" you will see these two messages in the console (with other messages in between printed over the rest of their code):
However, after the first crash of the web socket, no matter how many times you try to click on the button you will only get the first message (web client sends the request) but never will get the second message (request gets lost in the void):
Please note this corresponds to your behavior observed in VBA:
the script seems to get stuck after clicking on that button 3/4 times.
It doesn't get stuck, actually your script keeps on executing correctly. It's the website that times out.
I have tried to figure out why the web socket crashes, but no luck. It just seems a timeout (I've had this a lot more while debugging their JavaScript, so my breakpoints were causing the timeout) but I can't make you sure it's the only cause. Since you're not controlling the process between the web client and the web server, all you can do is to hope that it doesn't timeout.
Also, I believe using Selenium automatically sets some longer timeouts (because of the long execution time) and this somehow allows you to keep the web socket more tolerant with respect to the timeouts.
The only way I found to restore the connection after a crash of the web socket is completely reload the web page and restart the process from scratch.
My suggestions
I think you might go with building an XHR request and sending through JavaScript, because their API (through which the web client/web socket deliver the request to the web server) is pretty exposed in their front-end code.
If you open their file FinderAPI.js, you will see they've left the endpoints and API configurations harcoded:
var FinderAPI = {
store: null,
state: null,
finderEndpoint: '/api/v1/bsg/etp/finder/list',
bidAskEndpoint: '/api/v1/prices/bidAsk/get',
instrumentNameEndpoint: '/api/products/ProductTypeMapping/InstrumentNames',
nameMappingEndpoint: '/api/v1/bsg/general/namemapping/list',
apiConfig: false,
initialize: function initialize(store, finderEndpoint) {
var apiConfig = arguments.length > 2 && arguments[2] !== undefined ? arguments[2] : false;
this.store = store;
this.state = store.getState();
this.apiConfig = apiConfig;
this.finderEndpoint = finderEndpoint;
},
This means you know the URL to which you should send your POST request.
A request also requires a Bearer Token to be validated by the server. Lucky you, they have also forgot to protect their tokens providing (GORSH) a GET end point to get the token:
End-point: https://www.boerse-stuttgart.de/api/products
Response:
{"AuthenticationToken":"JgACxn2DfHceHL33uJhNj34qSnlTZu4+hAUACGc49UcjUhmLutN6sqcktr/T634vaPVcNzJ8sHBvKvWz","Host":"frontgate.mdgms.com"}
You'll just have to play around with the website a little bit to figure out what is the body of your POST request, then create a new XmlHttpRequest and send those values inside it to retrieve the prices directly in your VBA without opening the webpage and robotic-scraping.
I suggest you start with a breakpoint on the file FinderAPI.js, line 66 (the line of code is this.post(this.finderEndpoint, params), params should lead you to the body of the request - I remember you can print the object as string with JSON.stringify(params)).
Also, please note that they use a pagination of 50 results each time, even though their API supports up to 500 of them. In other words, if you get to sweep the value 500 (instead of 50) into their pagination property sent to the API for the request:
... then you will get 500 results per time instead of 50, so reducing by 10 the time your code will spend scraping the webpage in case you decide not to go deeper into the XHR solution.
Could you try to change
Do
Set post = Html.querySelector(".bsg-searchlist__load-more button.bsg-btn--juna")
If Not post Is Nothing Then
post.ScrollIntoView
post.Click
Application.Wait Now + TimeValue("00:00:05")
Else: Exit Do
End If
Loop
to:
Set post = Html.querySelector(".bsg-searchlist__load-more button.bsg-btn--juna")
If Not post Is Nothing Then
post.ScrollIntoView
While Not post Is Nothing
Debug.Print "Clicking"
post.Click
Application.Wait Now + TimeValue("00:00:05")
Wend
Debug.Print "Exited Click"
End If
(untested)
I currently have a SAPI voice implemented which works fine. I would like to know how do I handle Pause event in the engine when the application calls ISpVoice->Pause.
As near as I can tell, ISpVoice::Pause is implemented entirely in the SAPI layer and doesn't make any engine calls.
I'm trying to understand exactly what your end goal is, but I think I'll take a stab at answering this.
For voices, pause does the following according to MSDN:
ISpVoice::Pause pauses the voice at the nearest alert boundary and closes the output device, allowing access to pending speak requests from other voices.
Assuming you've used the speak call with SPF_AYSNC, you might also have to also set event interests and alert boundaries as seen here
Where the example shows handling bookmark alert boundaries
'Create a Normal Priority voice - male
Set objHIM = New SpVoice
Set objHIM.Voice = objHIM.GetVoices("gender=male").Item(0)
objHER.Speak "the priority of this voice is S V P alert"
objHIM.Speak "the priority of this voice is S V P normal"
objHIM.EventInterests = SVEBookmark 'Receive bookmark events only
objHIM.AlertBoundary = SVEPhoneme 'Let alert voices interrupt words
'Normal voice speaks text which generates events.
strSpeak = "This is text that contains bookmarks " _
& "for the purpose of generating events."
objHIM.Speak strSpeak, SVSFIsXML + SVSFlagsAsync
I am writing a program to control iTunes using the iTunes COM Interface and VB.NET. I am trying to figure out how to update labels in my program with the new track information when the track changes in iTunes. I haven't been able to figure this out. When the track changes, how do I update my labels with that new information? My program doesn't seem to know that the track changed. I try to update the information, but the program doesn't seem to get the new track information from iTunes. What do I need to do? I am using this method to update the labels:
Private Sub UpdateLabels()
Label1.Text = "Current Track: " + track.Name
Label2.Text = "Current Artist: " + track.Artist
Label3.Text = "Length: " + track.Time
End Sub
I think the labels are technically being updated, but as I said before, I don't think the program is getting the new track information, so it doesn't appear that they changed. If I restart my program, it gets the new information and updates the label accordingly.
The documentation is here.
I'm using the sample app from microsoft here:
http://msdn.microsoft.com/en-us/library/hh394041(v=vs.92).aspx
as a starting point to develop an app that allows the user to record multiple videos in the app to a collection.
What is the best way to accomplish this? I noticed that the example uses a fileSink object to specify the capture source and file name in isolated storage.
Private Sub StartVideoRecording()
Try
' Connect fileSink to captureSource.
If captureSource.VideoCaptureDevice IsNot Nothing AndAlso captureSource.State = CaptureState.Started Then
captureSource.Stop()
' Connect the input and output of fileSink.
fileSink.CaptureSource = captureSource
fileSink.IsolatedStorageFileName = isoVideoFileName
End If
' Begin recording.
If captureSource.VideoCaptureDevice IsNot Nothing AndAlso captureSource.State = CaptureState.Stopped Then
captureSource.Start()
End If
' Set the button states and the message.
UpdateUI(ButtonState.Recording, "Recording...")
' If recording fails, display an error.
Catch e As Exception
Me.Dispatcher.BeginInvoke(Sub() txtDebug.Text = "ERROR: " & e.Message.ToString())
End Try
End Sub
How would I then query that collection and allow the user to select that video for viewing in a listview? Is there no way to specify a folder to keep the video files organized?
Just looking for some advice on best practices to get this done. I wanted to use a video chooser that would allow the user to choose a video from their photo roll, but Windows Phone doesn't currently allow this.....
Well, is there any need to use a folder? noone can browse the ISO storage, so I see no need for folders :).
Using that tutorial, two files are created: "CameraMovie.mp4" and "CameraMovie.mp4.jpg" (the jpg is at least created on my phone, use ISETool to see the content of the ISO storage).
To record multiple videos, you would have to rename the filename each time
private string isoVideoFileName = "CameraMovie.mp4";
// reset the name when the recording starts
isoVideoFileName = DateTime.Now.ToString("yyyy-MM-dd_HH_mm") + ".mp4";
just change that variable when you start recording.
At the end of each recording, add the name of the video to a list, and once a video is added, you save the list (also to the ISO storage).
While loading the app, you load the list from the ISO, use the jpg's to make video tiles (or w/e you wanna do with it ;))
Hope this will help you a little forward, if not you have found the solution yet.
note, im sorry for using C#, whereas you use VB. It is however that i dont know VB well enough to type it at will ;)
In HP Qualtiy Center,
I have a test case where I have attached some documents.
I want to call directly the attachment from other test cases(not the test case).
Can somebody help out?
You can define a custom action (which will appear as additional button), and use OTA API to retrieve attachments you want when user clicks on that icon.
(it's been a while since I worked with QC workflow, so apologies for possibly wrong syntax, but it demonstrates the idea)
Add new action button through UI (let's call it "getMyFiles"). After that catch event - user clicked the button:
Function ActionCanExecute(Action)
...
if Action = "getMyFiles"
getMyFiles
end if
...
End Function
Now retrieve the attachments and do whatever you want with them (e.g. open...copy...save somewhere)
Sub getMyFiles
Set tst = TDConnection.TestFactory.Item(##id of the test with attachments##)
Set attf = tst.Attachments
' From here you can do whatever you want with those attachments
' In my example I will just download them:
Set attLst = attf.NewList("")
For Each att In attLst
' Don't remember what those parameters mean (took from old example),
' so check OTA API guide
att.Load True, ""
Next
End Sub
That's about it