I am trying to scrape from a fashion website that uses Javascript, using Scrapy
This is the page: https://www.thekooples.com/us_en/women/ready-to-wear/dresses.html
I have docker, and followed the instructions on splash docs to set up splash on localhost:8050.
I am able to render https://quotes.toscrape.com/js/ properly.
As I understand, that is a js page, and I disabled js, and it does look different when I do so.
However, I am unsuccessful in rendering the fashion webpage. This is what I get:
That is actually how the page looks like without js, so I know it is unsuccessful. What could be happening?
You can do try the following two things:
Increase the delay. You can also find an example script in splash server's homepage where you can wait for a particular element to appear.
You can download and print the HAR and see if any of the requests failed. If it failed then you may need to add some user-agent in your splash request.
Related
I have a url https://www.dummyvideo.com/ As soon as this url is hit, there is a single video which starts playing. When I check on browser, segment is not showing in response. Same is the case in JMeter. I would like to know which protocol should I use for acfor achieving this goal. Also, how do I conclude that if my script is working as I dont see segment response on browser as well as on JMeter.
As per JMeter project main page:
JMeter is not a browser, it works at protocol level. As far as web-services and remote services are concerned, JMeter looks like a browser (or rather, multiple browsers); however JMeter does not perform all the actions supported by browsers. In particular, JMeter does not execute the Javascript found in HTML pages. Nor does it render the HTML pages as a browser does (it's possible to view the response as HTML etc., but the timings are not included in any samples, and only one sample in one thread is ever displayed at a time).
The video is being "played" via JavaScript function call and JMeter doesn't automatically "play" the video as it's not capable of executing JavaScript
Since your URL doesn't lead to the website with the video I can only give general recommendations:
If it's possible to find out the exact URL of the video from the page source you can extract it using a suitable JMeter Post-Processor and add another HTTP Request sampler which will download the video
If the video is in form of M3U playlist you can use JMeter HLS Plugin which parses the playlist and gets media streams out of it
In both cases make sure to add a relevant Timer to simulate the user watching the video before proceeding to next page/iteration
Im trying to scrape a site that requires logging in. so i login.
The pages directly after the login seem to have alot of javascript.
How can i test out the responses of splash on scrapy shell?
Once i login, how can i run splash on the next url via splash command line and have it process the javascript and give me a response i can parse?
I dont understand what i actually need to do to run the splash service and how it functions with scrapy... Please point me in the right direction.
I have created a sample application with login page and home page.
i have loaded the login page as Main Page successfully and i have homePage.html/homePage.js files. What i need to do is, when i click on Login button from loginPage.html file, user should navigate to Home Page. I couldn't see any APIs fro navigate/load to different html files.
i referred some worklight samples it says loading on same HTML using div and pageport commands.
$("#pagePort").load("pages/homePage.html", function(){
alert("Home Page Loaded")
});
Can any one help me to get resolve, how we can load another html on click of login button?
Worklight is a single DOM application, hence the notion of "pages" is an artificial one.
You have 1 singular HTML file and what you need to do is replace content inside it to display different content, or "pages".
There are numerous ways you can accomplish this. None of them relate directly to Worklight, but to the framework you select to work with, be it jQuery, jQuery Mobile, Dojo, Ionic and so on...
The important thing to remember is to never navigate away from the index.html file because you will then "lose" the Worklight context and your application will then fail loading. The reason is because from the index.html file, you load various required Worklight-related files.
In this Worklight 6.2 project I demonstrate, using jQuery Mobile, how to navigate between pages. For your purpose, think of them as a mainpage that is a login page and then some other page that is the home page...
Related:
https://stackoverflow.com/a/25060136/1530814
https://stackoverflow.com/a/22171687/1530814
I am developing the WP HTML5 app. I've set of HTMLs, js and css created for it.
The IsScriptEnabled flag is set to true, before I Navigate to the HTML page added as project asset.
The js is not getting executed from the index.html page, I am navigating to. When trued, the same js execution is happening properly on iOS and Android.
What can be the possible reasons for the same.
Anything that has to be done (apart from enabling script) before navigating to the page? Please help!
The IsScriptEnabled flag does not enable/disable JS, but it enables JS to call out to the app and vice versa.
So, I assume you have some JS in the OnLoad of the page you are navigating to. If you replace your code with a simple document.writeLine("test"), does that work? If it does work I'd assume that IE's JS engine is (once again) not compliant.
You could also listen to window.onError
We're trying to validate if a URL fires upon loading a web page. Is there a way to do this programmatically using Selenium RC? The event does not appear within the page HTML or DOM.
Thanks.
Can you specify the URL as part of your test?
If so, then what about running a httpd server and checking if it gets a hit when your page loads?
The selenium.start("captureNetworkTraffic=true"); enables the 'capture' of all the HTTP requests/responses associated with the loading of the web page. Once formatted, these results can be reported easily.