I was running a python script responsible for web scraping some pages using multiprocessing techniques.
In order to wait for a page to get fully loaded, I used the method set_page_load_timeout() set to 30s.
I put a driver.get() inside a try-except structure.
However, I observed that in my case, when all the 5 Chrome instances are busy with 5 different pages, the next page pulled from the task queue goes to my exception and stored in my error file. I presume this occurs since the Chrome instances are still trying to load the 5 previous pages and not able to redirect to this new page.
How could I get over this issue?
Related
I'm running a website using Typo3-10.
The page actually works, but on every 3rd or 4th call to the page, it's slow as hell and time to first byte takes up to 10 seconds.
It's intermittently and not bound to a specific page.
Caching is enabled (although most of the content is not cacheable, since it's dynamic content that changes constantly).
There are no errors when the page is loaded (at least, the log is empty [and yes, logging is enabled]).
I have no idea where to start or what to look for…
Some context:
We have a nextjs app that renders a page to HTML (using next's renderToHTML). We use a Puppeteer browser page to create a PDF from the HTML, which is then returned from the service running the nextjs app.
The pages that are rendered using renderToHTML contain one or more queries that are executed using Apollo.
The problem
We are experiencing a problem where old queries are being re-run when the app generates a new page.
For example: if there are 3 requests one after another, the first request will execute the queries needed to generate the page being requested; the second will execute both the queries of the first and the second page; and the third will execute the queries of all three pages.
We have measured the incoming requests to the PDF service, and how often (and for which pages) that renderToHTML is called, and they are both working correctly (meaning that the service is not being spammed with requests which would cause the repeated queries).
The issue is that 1 request, generates 1 page, but n Apollo queries are executed, for both the current and a number of previous requests.
It seems like a some kind of caching problem, where the pages/requests are being cached and re-executed when a new request comes in, but we aren't sure if it is next or puppeteer that is causing the problem.
Has anyone encountered this/a similar problem? Any information that anyone has about this issue would be very greatly received!
Versions
"next": "10.0.0" (the issue started with "9.5.3", we tried to update next to see if it solved the problem)
"#apollo/client": "3.2.2"
"puppeteer": "5.4.0"
Website I am testing needs to load libraries from xxxx server.
Sometimes it takes to long and website display information Waiting for xxxx (I am using Chrome WebDriver).
Is it possible to treat it as an error and catch it using Selenium?
Sometimes when it takes to long and you see website display information Waiting for xxxx it is not actually an error. It means the the JavaScripts and the AJAX Calls associated with the Website are getting executed while the Page Loads. Once the JS & AJAX Calls gets executed completely, the Page Loading completes.
Now as per your question Is it possible to treat it as an error and catch it? the Answer is Yes.
This can be achieved by using the pageLoadTimeout method which Sets the amount of time to wait for a page load to complete before throwing an error.
Defined as pageLoadTimeout(long time, java.util.concurrent.TimeUnit unit), an example will be as follows :
driver.manage().timeouts().pageLoadTimeout(2, TimeUnit.SECONDS);
The error thrown is as follows :
Exception in thread "main" org.openqa.selenium.TimeoutException: Timeout loading page after 2000ms
You can find a detailed implementation of pageLoadTimeout in pageLoadTimeout in Selenium not working discussion.
I am using Java and Selenium to write a test. Many times in my test i have to wait till the page or some part of it loads something. I want to wait till the load is finished. I found these two:
1) pageLoadTimeout
2) setScriptTimeout
But I don't know the exact difference between them or even if they are what I need.
The document says:
setScriptTimeout: Sets the amount of time to wait for an asynchronous script to finish execution before throwing an error. If the timeout is negative, then the script will be allowed to run indefinitely.
and
pageLoadTimeout: Sets the amount of time to wait for a page load to complete before throwing an error. If the timeout is negative, page loads can be indefinite.
but I don't really get it.
what I need is a wait that stops the test till nothing is happening in the web page anymore.
Define "nothing is happening in the web page." Now define it in such a way that will satisfy every web page ever created, or that ever will be created. Did your definition include that no more network traffic is coming over the wire? Did your definition include that the onload event has fired? How about document.readyState returning a value meaning "complete?" Did you take JavaScript execution into account? How about JavaScript scheduled using window.setTimeout()? WebSockets? Service Workers? What about <frame> or <iframe> elements?
Now you know why WebDriver doesn't have a "wait for page complete" type of wait. The items mentioned above don't even begin to scratch the surface of the list of things a user could legitimately use in a definition of "nothing is happening on the page anymore." The correct approach would be to wait for the element you want to interact with on the next page to be available in the DOM. The WebDriverWait class is ideally suited for exactly this.
Through selenium webdriver I'm attempting to automate a file upload (of large files consisting of hundreds of megabytes).
Usually it works fine, but sometimes the site doesn't upload the file and the upload progress gets stuck at 0%.
Therefore I'd like to check if the progressbar style.width is 0% after a certain interval of time (in order to determine if it's stuck).
Usually this would be no problem, however apparently Selenium is blocked when uploading a file (I assume it's because the request is in progress) and thus I get a timeout exception if I attempt to check the width before the upload has finished.
Do anyone know a way to solve this issue? IE. ignore the block in selenium to allow me to check the DOM without getting a timeout exception.
Try using 'WebDriverEventListener' class which has methods that can be used to
track file-uploading.