Testing a live website with QUnit - testing

Can I test live websites using QUnit? For example, can I write a test that says:
Go to google.com
Enter a search term
Click 'Google Search'
Check there are 10 results and 2 ads
Would QUnit be an appropriate tool for this kind of "live" testing?

You can achieve that using qunit if qunit is the only tool/testing framework that is available for you and the page you will be testing allows GET requests.
How to do that qunit is you can make an ajax call to the page you are testing using JSONP and get the response. Then you would assert certain elements on the response exists.
For google, google itself has a very complex page structure on search results, I would not even attempt to do anything like this.
I would use qunit for testing javascript components on their own without dependencies.
If you are looking for another tool to do this task, I would recommend Selenium, which would do exactly what you want.
Good luck.

Do you want to test a website you own or a random live website?
If you want to test your own website
you can embed the live site in a iframe and perform actions in the user interface in your tests.
If you want to test live websites like google.com you need to do this server side since you can't access them from javascript/QUnit.
When you where the owner of a site like google.com you could do:
var submitted = false;
function starttests(){
if(!submitted)
test("testInput", function() {
expect(1);
submitted = true;
var dom = iframe.contentWindow || iframe.contentWindow.document;
jQuery(dom).find('input[type=text]').val("Testing google.com");
jQuery(dom).find('form').submit();
ok( true, "form submitted" );
});
else
test("testResult", function() {
var dom = iframe.contentWindow || iframe.contentWindow.document;
// Check for elements in dom.
});
}
iframe.onload(starttests);
iframe.src = "http://google.com";

Related

Scrape dynamic websites using dart in flutter app

I have a website that generates a list of items using some javascript and I am trying to scrape it inside my flutter app using beautiful soap package for dart. The thing is that I am unable to scrape the dynamic data generated by the java script. I want to implement a solution that allows me to grab the source code of the website after it fully loads inside the app. A hidden webview inside the app would be perfect but what is blocking me is that how to get the data after the webview loads. This is my main concern. Code examples would be appreciated. Also better practices are welcomed.
What I have came to realize is that scraping dynamic websites that contain some javascript or a website that you want to click in it using a certain script to scrape it properly is not possible over flutter mobile. What you should do is to move the scraping to the cloud by creating your own api then using this api to return the response to your app. This will make scraping easier since you will not have to update your app for every error you find in your scripts. Also imagine that the website that you target updates itself every week, then you will have to update your app every week and wait for approval from all the stores you are subscribed to. A simple example would be using cloud functions from firebase in combination with javascript by utilizing the puppeteer package. A simple video tutorial is here: Tutorial over youtube
After lots of research I did indeed find a way.
Basically loading a hidden webview and scraping the data off of it, then showing it on screen. Here's how..
Spinning a webview in the UI
The Visibility widget and width/height properties will make sure the webview is impossible to be seen by the user. I suggest showing a loading screen until the data is scraped.
Visibility(
visible: false,
maintainState: true,
child: Container(
height: 1,
width: 1,
child: WebViewPlus(
onWebViewCreated: (controller) async {
log.e("onWebViewCreated");
await model.onWebViewCreated(controller);
},
onPageFinished: (url) async {
log.e("onPageFinished");
await model.onPageFinished(url);
},
javascriptMode: JavascriptMode.unrestricted,
),
),
),
The actual scraping
onWebViewCreated(controller) async {
this.webViewController = controller;
// Load the URL
await controller.loadUrl("<Your Website URL Here>",headers:_apiService.getAuthHeader());
// Get the HTML of the webpage as a JSON object
String docu = await webViewController?.webViewController.evaluateJavascript('document.documentElement.innerHTML') as String;
// Convert from JSON to String
var jsonString = json.decode(docu);
// Parse the String to a HTML DOM to actually access the elements
var dom = parse(jsonString);
// Some logic I needed in my application by scraping
for (var child in dom.getElementById("autodl-log-tbody")!.children) {
feed.add(child.text);
}
}
Pro Tip : If you think the webpage might need a bit more time to load, you can stall the execution of the function by using await Future.delayed(Duration(Seconds:5)); before the line of code where you load the URL in the onWebViewCreated() function.

Phantomjs and ajax

I am a total noob with phantomjs, and I don't do any web development.
However, I want to get phantomjs to sit on my page, like a browser.
When a browser does this, I see a call every few seconds (5?). It appears to be an ajax thing.
When I load the page with phantomjs, it loads successfully, but I never see the call that is made as if I am in a browser.
Hopefully, this is not a difficult thing. Since I do not have any experience I don't want to have to jump through a bunch of hoops to make it work!
Thanks!
My very basic code is such:
var webPage = require('webpage');
var page = webPage.create();
page.open('http://myurl.com/', function(status) {
console.log('Status: ' + status);
// Do other things here...
});

How to open a page with Phantomjs without running js or making subsequent requests?

Is there a way to just load the server generated HTML (without any js or images)?
The docs seem a little sparse
The strength of PhantomJS is exactly in its ability to emulate a real browser, which opens a page and makes all the subsequent request. If you want just html maybe better use curl or wget?
But nevertheless there is a way not to run js or load images: set corresponding page settings: http://phantomjs.org/api/webpage/property/settings.html
page.settings.javascriptEnabled = false;
page.settings.loadImages = false;

Communication Between WebView and WebPage - Titanium Studio

I am working in a Mobile project (using Titanium Studio), in which i have the below situation
1) My Mobile app contacts Rails backend to check some data, say check validity of a
user id.
2) I found a way to load web pages in Mobile app, i.e., WebView
3) I could able to load the desired url, ex http://www.mydomain.com/checkuser?uid=20121
which would return data like status:success
But i need to read this data to show whether the response from server is a success or failure, how do i achieve this?
NOTE : The above mentioned scenario is an usecase, but actually what happens is i load a third party url in WebView and when user enters the data and submits, the result will be posted back to my website url.
EDIT : So the process is like below
1) WebView loaded with third party url like http://www.anyapiprovider.com/processdata
2) User will enter set of data in this web page and submits the page
3) The submitted data will be processed by the apiprovider and it returns data to my web page say http://www.mydomain.com/recievedata
This is the reason why i am not directly using GET using HTTPClient
FYI : I tried to fire Ti.APP events right from the actual web page as suggested by few articles, but most of them says this will work only if the file loaded is in local and not a remote file. Reference Link
Please suggest me if my approach has to be improved.
Thanks
If you don't want to follow Josiah's advice, then take a look at the Titanium docs on how to add a webview.addEventListener('load',... event listener and use webview.evalJS() to inject your own code into the third party HTML.
Maybe you can inject code to trap the submit event and fire a Ti event to trigger the downloading of data from your website.
Communication Between WebViews and Titanium - Remote Web Content Section
I found a solution for my problem
1) Load the http://www.mydomain.com/checkuser?uid=20121 in a webview
2) Let user enter and submit data to third party url
3) Recieve the response from third party url and print only <div id="result">status:success</div> in http://www.mydomain.com/recievedata page.
4) Add event listener for the web view as follows
webView.addEventListener('load', function(data)
{
//Add condition to check if the loaded web page has any div with id = result (to check if this is /recievedata page)
alert(webView.evalJS("document.getElementById('result').innerHTML"));
});
The above alert would print the result status:success, read it in webview load event
and take actions in web accordingly.
It works fine for me.
Instead of loading it in a WebView why not just GET it using a HTTP Client? This is much cleaner, and more standards based:
var xhr_get = Ti.Network.createHTTPClient({
onload : function(e) {
// Here is your "status:success" string
var returnValue = this.responseText;
},
onerror : function(e) {
Ti.API.info(this.responseText);
Ti.API.info('CheckUserProgressOnActivity webservice failed with message : ' + e.error);
}
});
xhr_get.open('GET', 'http://www.mydomain.com/checkuser?uid=20121');
xhr_get.send();

Screen Scraping - still not working

I have browsed through many posts on this and have tried some of the suggestions but still not understanding it fully.
I would like to scrape html pages that have some script running that usually executes the script to display a link after clicking. Some mentioned firebug and others talked about reverse engineering the code I need. But after trying reverse engineering I still dont see how to get the data after tracing the script function.
jQuery('.category-selector').toggle(
function() {
var categoryList = jQuery('#category-list');
categoryList.css('top', jQuery(this).offset().top+43);
jQuery('.category-selector img').attr ('src', '/images/up_arrow.png');
categoryList.removeClass('nodisplay');
},
function() {
var categoryList = jQuery('#category-list');
jQuery('.category-selector img').attr('src', '/images/down_arrow.png');
categoryList.addClass('nodisplay');
}
);
jQuery('.category-item a').click(
function(){
idToShow = jQuery(this).attr('id').substr(9);
hideAllExcept(jQuery('#category_' + idToShow));
jQuery('.category-item a').removeClass('activeLink');
jQuery(this).addClass('activeLink');
}
);
I am using vb.net and some sites were easy using firebug where looking at the script I was able to pull the data that I needed. What woudl I do in this scenario? the link is http://featured.typepad.com/ and the categories are what I am trying to access. Notice the url does not change.
Appreciate any responses.
My best suggestion would be to use Selenium for screen scraping. It is normally used for automated website testing but would fit your case well. I've used to screen scrape AJAX pages on multiple occasions where the page was heavily Javascript dependent.
http://seleniumhq.org/projects/ide/
You can write your screen scraping code to run in .NET and it can use Firefox or IE to run your screen scraping with.
With selenium what you'll do is record a screen scraping session with the Selenium IDE in Firefox (look for the Firefox extension in the link above). That screen scraping session can either output an HTML template or C# code. It might be able to output VB as well.
You'll copy the C# or VB.NET output from the screen scrape into a selenium .NET project that you'll create and then run the Selenium project through Nunit.
I'd suggest looking online for some help with getting Selenium started and working but this should get you on your way.