We are trying TestCafe with 50 concurrency and TestCafe randomly with "Cannot read property 'stackFrames' of undefined"
Same code base when we run with 20 threads works fine without any issues. Is there any upper limit on number of threads in Testcafe?
Note: - We have enough cpu to support 50 threads in aws (c5a.8xlarge)
TestCafe has no formal limit on the number of threads. However, we have not tested the concurrency mode with so many threads. Therefore, I can only give general recommendations: make sure that you have enough resources to run so many browsers (not only CPU): RAM, disk I/O, GPU. Also, run browsers in headless mode, for example: testcafe -c 50 chrome:headless test.js.
Related
I successfully implemented a solver that fits my needs. However, I need to run the solver on 1500+ different "problems" at 0:00 precisely, everyday. Because my web-app is in ruby, I built a quarkus "micro-service" that takes the data, calculate a solution and return it to my main app.
In my application.properties, I set:
quarkus.optaplanner.solver.termination.spent-limit=5s
which means each request take ~5s to solve. But sending 1500 requests at once will saturate the CPU on my machine.
Is there a way to tell OptaPlanner to stop when the solution is good enough ? ( for example if the score is stable ... ). That way I can maybe reduce the time from 5s to 1-2s depending on the problem?
What are your recommandations for my specific scenario?
The SolverManager will automatically queue solver jobs if too many come in, based on its parallelSolverCount configuration:
quarkus.optaplanner.solver-manager.parallel-solver-count=3
In this case, it will run 3 solvers in parallel. So if 7 datasets come in, it will solve 3 of them and the other 4 later, as the earlier solvers terminate. However if you use moveThreadCount=2, then each solver uses at least 2 cpu cores, so you're using at least 6 CPU cores.
By default parallelSolverCount is currently set to half your CPU cores (it currently ignores moveThreadCount). In containers, it's important to use JDK 11+: the CPU count of the container is often different than from the bare metal machine.
You can indeed tell the OptaPlanner Solvers to stop when the solution is good enough, for example when a certain score is attained or the score hasn't improved in an amount of time, or combinations thereof. See these OptaPlanner docs. Quarkus exposes some of these already (the rest currently still need a solverConfig.xml file), some Quarkus examples:
quarkus.optaplanner.solver.termination.spent-limit=5s
quarkus.optaplanner.solver.termination.unimproved-spent-limit=2s
quarkus.optaplanner.solver.termination.best-score-limit=0hard/-1000soft
I read that the --runInBand flag speeds up Jest test duration by 50% on CI servers. I can't really find an explanation online on what that flag does except that it lets tests run in the same thread and sequentially.
Why does running the test in the same thread and sequentially make it faster? Intuitively, shouldn't that make it slower?
Reading your linked page and some other related sources (like this github issue) some users have found that:
...using the --runInBand helps in an environment with limited resources.
and
... --runInBand took our tests from >1.5 hours (actually I don't know how long because Jenkins timed out at 1.5 hours) to around 4 minutes. (Note: we have really poor resources for our build server)
As we can see, those users had improvements in their performances on their machines even though they had limited resources on them. If we read what does the --runInBand flag does from the docs it says:
Alias: -i. Run all tests serially in the current process, rather than creating a worker pool of child processes that run tests. This can be useful for debugging.
Therefore, taking into consideration these comments and the docs, I believe the improvement in performance is due to the fact that now the process runs in a single thread. This greatly helps a limited-resource-computer because it does not have to spend memory and time dealing and handling multiple threads in a thread pool, a task that could prove to be too expensive for its limited resources.
However, I believe this is the case only if the machine you are using also has limited resources. If you used a more "powerful" machine (i.e.: several cores, decent RAM, SSD, etc.) using multiple threads probably will be better than running a single one.
When you run tests in multi-threads, jest creates a cache for every thread. When you run with --runInBand jest uses one cache storage for all tests.
I found it after runs 20 identical tests files, first with key --runInBand, a first test takes 25 seconds and next identical tests take 2-3s each.
When I run tests without --runInBand key, each identical test file executes in 25 seconds.
By default, Bazel runs tests in a parallel fashion to speed things up. However, I have a resource (GPU) that can't handle parallel jobs due to the GPU memory limit. Is there a way to force Bazel to run tests in a serial, i.e., non-parallel way?
Thanks.
--jobs 1 will limit the number of parallel jobs Bazel runs to 1.
You can also modify the test targets and add tags = ["exclusive"] to prevent specific test to run in parallel (see http://bazel.io/docs/test-encyclopedia.html).
Use --local_test_jobs=1 to only run a single test job at a time locally.
The max number of local test jobs to run concurrently. Takes an integer, or a keyword ("auto", "HOST_CPUS", "HOST_RAM"), optionally followed by an operation ([-|]) eg. "auto", "HOST_CPUS.5". 0 means local resources will limit the number of local test jobs to run concurrently instead. Setting this greater than the value for --jobs is ineffectual
tags = ["exclusive"] has other complications to consider with respect to caching.
--jobs will serialize the entire build process, not just testing, so it's less than ideal.
There are 2 resources Bazel will respect limitations upon: RAM and CPU. You may hijack one (Probably RAM) to represent GPU(s) as they're available to a run and required by a test. (I've stopped short of doing this for a limited hardware resource because it feels to inelegant, but I can't think of a reason it shouldn't work.)
Future releases of Bazel should support extra resources like GPUs
and releases that contain that change should support extra resource tags like "resources:GPU:1" when --local_extra_resources=gpu=1 is set. This should enable GPU tests to be bound by a limited quantity of GPUs, and for them to run non-exclusively and without limiting the total number of --jobs or "test_jobs"
I got a requirement from my client to implement Selenium grid. Requirement details as follows:
Number of nodes running concurrent test cases: 200-250
3 Selenium nodes per machine (~ 66 to 85 machines running the 200-250 nodes)
Every machine is a Windows machine and has 8GB RAM
The test cases are long-running in nature. about 10000 steps per test case. (click, type etc are considered to be the test steps)
The option of using grid services from providers like browserstack is ruled out. I will have to do the grid setup locally.
I am looking for answers for questions like, Has anyone tried grid setup with such level of complexity? Are there any limitations of Selenium to support such large setup? Would there be performance impact due to excessive context switches between threads? What problems should I be prepared to face?
Thank you for your help.
It seems possible to me. Some things to consider:
Long running tests with many steps should be broken down into
smaller test units.
Each grid node, if they have 8GB of ram and dual-proc, should be able to handle up to 10+ browsers, not just 3.
If the tests are simple, you could use PhantomJS 2.x headless browser and get more speed and throughput out of your tests. This
is especially useful if the version of the browser your testing with
doesn't much matter, such as with verification tests. Regression
tests, on the other hand, need specific browser versions to verify
regressions caused by code.
I would say that a grid hub with many nodes on it , should be able to handle 50+ threads at a level of 8gb and dual proc. So, for
each 50 threads you would want 5 nodes and 1 hub.
Thats just some thoughts...
I am using webkit based tools to built a headless browser for crawling webpages (I need this because I would like to evaluate the javascript found on pages and fetch the final rendered page). But, the two different systems I have implemented so far exhibit very poor performance. I have implemented two different systems, both of which use webkit as the backend:
Using Google Chrome: I would start Google Chrome and communicate with each tab using webSockets exposed by Chrome for remote debugging (debugging over wire). This way I can control each tab, load a new page and once the page is loaded I fetch the DOM of the loaded webpage.
Using phantomjs: phantomjs uses webkit to load pages and provides a headless browsing option. As explained in the examples of phantomjs, I use page.open to open a new URL and then fetch the dom once the page is loaded by evaluating javascript on the page.
My goal is to crawl pages as fast as I can and if the page does not load in the first 10 seconds, declare it failed and move on. I understand that each page takes a while to load, so to increase the number of pages I load per second, I open many tabs in Chrome or start multiple parallel processes using phantomjs. The following is the performance that I observe:
If I open more than 20 tabs in Chrome / 20 phantomjs instances, the CPU usage rockets up.
Due to the high CPU usage, a lot of pages take more than 10seconds to load and hence I have a higher failure rate (~80% of page load requests failing)
If I intend to keep the fails to less than 5% of the total requests, I cannot load more than 1 URL per second.
After trying out both the webkit based systems, it feels like the performance bottleneck is the webkit rendering engine and hence would like to understand from other users here, the number of URLs per second that I can expect to crawl. My hardware configuration is:
Processor: Intel® Core™ i7-2635QM (1 processor, 4 cores)
Graphics card: AMD Radeon HD 6490M (256MB)
Memory: 4GB
Network bandwidth is good enough to be able to load pages more than the performance that I am observing
The question I am trying to ask this mailing list is, does any one have experience using webkit for crawling web pages for a random set of URLs (say picking 10k URLs from twitter stream), how many URLs can I reasonably expect to crawl per second?
Thanks
This question is actually more related to hardware than the software but let me point you in some better directions anyway.
First, understand that each page is itself spawning multiple threads. It will download the page, and then start spawning off new download threads for elements on the page such as javascript files, css files and images. [Ref: http://blog.marcchung.com/2008/09/05/chromes-process-model-explained.html ]
So depending on how the page is structured, you could end up with a fair number of threads going at the same time just for the page, add on top your trying to do too many loads at once and you have a problem.
The Stack Overflow thread at Optimal number of threads per core gives further information on the situation you are experiencing. Your overloading your cpu.
Your processor is 4 physical 8 logical cores. I would recommend spawning no more than 4 connections at one time, leaving the secondary logical cores to handle some of the threading there. You may find that you even need to reduce this number but 4 is a good starting point. By rendering pages 4 at a time instead of overloading your whole system trying to render 20 you will actually increase your overall speed since you end up with far less cache swapping. Start by clocking your time against several easily timed locations. Then try with less and more. There will be a sweet spot. Note, the headless browser version of PhantomJS is likely going to be better for you as in headless mode it probably won't download the images (a plus).
Your best overall option here though is to do a partial page rendering yourself using the webkit source at http://www.webkit.org/. Since all it seems you need to render is the html and the javascript. This reduces your number of connections and allows you to control your threads with far greater efficiency. You could in that case then create an event queue, spool all your primary url's into there. Spawn off 4 worker threads that all work off of the worker queue, as they process a page and need to download further source they can add those further downloads to the queue. Once all files for a page are downloaded into memory (or disk if your worried about ram) for a particular url, you can then add an item into the event queue to render the page and then parse it for whatever you need.
Depends on what data you are trying to parse, if you only care that javascript and the html , then hypertext query langage would offer an immense speedup, http://htql.net/ , or you can look to setting up something in the cloud such as http://watirmelon.com/2011/08/29/running-your-watir-webdriver-tests-in-the-cloud-for-free/