I'm having some trouble scraping some HTML that I'm getting from a postback on a site. It is an aspx page that I am trying to get the generated HTML from.
I have looked at the cookie data and session data and forum data being sent with Chrome developer tools and I still cannot get the page to respond with the search results despite mimicking almost all of it in my code.
There are 3 dropdowns on the page, 2 of which are pre-populated when you first visit the page. After choosing values for the first 2 (it does a postback every time you select on those two), it will populate values for the 3rd drop down. Once selecting a value in the 3rd drop down, you hit the search button and the results come back in a table below that.
After hitting the search button and getting the results on the screen, I went into developer tools and grabbed all of the values that looked relevant (especially all form values) and captured them in my code, but still no luck. Even captured the big viewstate exactly.
Here is a code sample of many code samples that I've tried. Admittedly, I'm not very familiar with some of these classes and I've been trying different code snippets.
I'm not sure if I'm doing it wrong in my code or if I'm just missing form data or cookies to make it execute the POST and return the correct data. My code currently returns HTML from the page back to the responseInString variable, but the HTML looks like it's the first version of the page (as if you visited it for the first time) with no drop down boxes selected and the 3rd is not populated with any values. So I don't know if my code is actually hitting the code-behind and doing the form POST to make it return data.
Any help would be greatly appreciated. Thank you!
using (var wb = new WebClient())
{
var data = new NameValueCollection();
data["_EVENTTARGET"] = "";
data["_EVENTARGUMENT"] = "";
data["_LASTFOCUS"] = "";
data["_VIEWSTATE"] = "(giant viewstate)";
data["__VIEWSTATEGENERATOR"] = "D86C5D2F";
//3 more form input/select fields after this with values corresponding to the drop downs.
wb.Headers.Add(HttpRequestHeader.Cookie,
".ASPXANONYMOUS=(long string);" +
"ASP.NET_SessionId=(Redacted);" +
" _gid=GA1.2.1071490528.1676265043;" +
"LoginToken=(Redacted);" +
"LoginUserID=(Redacted);" +
"_ga=GA1.1.1195633641.1675746985;" +
"_ga_38VTY8CNGZ=GS1.1.1676265043.7.1.1676265065.0.0.0");
wb.Headers.Add("Sec-Fetch-Dest", "document");
wb.Headers.Add("Sec-Fetch-Mode", "navigate");
wb.Headers.Add("Sec-Fetch-Site", "same-origin");
wb.Headers.Add("Sec-Fetch-User", "?1");
wb.Headers.Add("Content-Type", "application/x-www-form-urlencoded");
var response = wb.UploadValues("(the web page url)", "POST", data);
string responseInString = Encoding.UTF8.GetString(response);
return responseInString;
}
If clarification is needed, please let me know. If it can't be done, please let me know this as well. I am desperately trying to figure this out still
I was following Google's Dev guide to the Share button at the following site:
https://developers.google.com/+/web/share/
and I can not seem to figure out how, if it's even possible, to use a custom icon AND use the current URL instead of having to specify a URL.
I found this section of their site that specifies an anchor tag address:
"https://plus.google.com/share?url={URL}"
This would allow me to use a custom icon (and the only way I can use a custom icon as far as I can tell) and a few other custom parameters as well. But it looks like this method requires a specified URL and, as far as I can tell, provides no method to dynamically create the link depending on the current page.
If I use the code generator at the top, it will use the current page, but it calls on a Google hosted Java Script and in addition, it is a hover link that pops up when I hover over the icon. And of course, I also can't use a custom icon with the generator either.
I've been Googling every search term I could think of and searching this site as well and I haven't been able to find anyone else asking this question as of yet. I figured after about 20-30 minutes of searching that I wasn't going to find my answer via searching, so i apologize if this has been answered.
Just some background on my experience to give an idea of where I sit: I have a decent grasp of the workings of HTML and CSS. Javascript, however, I understand very very basic theory and that's about it. I definitely intend to learn, however, as it will prove a very valuable skill.
Thank you very much!!
I think I know what needs to be done, but...I don't know how to do it (or if it would even work) :|
my share link needs to link to a script that looks at the current page's URL, and then takes that information, and creates a dynamic link from it that will take the user to the following link: https://plus.google.com/share?url={URL from query will be here}.
I think that might work...it sounds like it would. Any thoughts? If so, any simple scripts around that would do just this?
Double thanks!!
--I finally found something that works, but it uses Javascript and I don't fully understand it, just enough to tweak it. It took me forever to find this, but it works with Google Plus, Facebook, or Twitter! (and I'm sure it will work with any other website that supplies a Share Link that requires a specified URL)
Here it is, I'm still looking for a better solution, but this does exactly what I was looking for:
<a href="javascript:(
function(){
var w=480;var h=380;
var x=Number((window.screen.width-w)/2);
var y=Number((window.screen.height-h)/2);
window.open('https://plus.google.com/share?url='+encodeURIComponent(location.href)+'
&title='+encodeURIComponent(document.title),'','width='+w+',height='+h+',left='+x+',top='+y +',
scrollbars=no');
})();" style="background: url(/wp-content/themes/HTML5/images/googleplus.png) no-repeat scroll left center transparent;">
Share to Google+</a>
EDIT! After spending some months learning Javascript, I've built a solution that is much better than that which is provided below. I'll leave my original answer, however, I want to place this better solution at the top.
This solution should work on ANY social media platform that gives you a custom share URL (that is to say, a url that allows you to manually type in an address to share).
Here is how it all works (and if anyone has any suggestions or tweaks that have more experience with JS, please let me know).
I assign variables to the document.URL and document.titleproperties.
I write a named function (I called mine, socialShare) that is set to run via an anonymous function on the window.onloadevent.
The socialShare function assigns variables to the location of my social button's within the HTML. In my case, I used IDs to locate the elements. The purpose of these variables is purely for aesthetics (I use these variables to re-write the the HTML code dynamically, so that when you hover over the share button, it displays the correct URL for sharing the current page you are on)
var fbShare = document.getElementById("fbShare");
var gplusShare = document.getElementById("gplusShare");
twitterShare = document.getElementById("twitterShare");
I then write three separate anonymous functions, one for each social media platform. Each function has two statements. The functions work as follows: the first part is the variable assigned to the location of the HTML element with the ID fbShare. The second part tells it to run the function when that element is clicked; .onclick. The third part is the anonymous function that will run when that element is clicked. The first statement of this function will open a new window; window.open; and in that new window, it will open the URL that is specified by feeding the window.open method parameters. The parameters are as follows (URL,name,specs) where URL is the URL you want to share, name is optional and left blank as seen by the empty set of quotes, and finally specs is where you specify attributes of the window (IE: width and height). The first parameter, the URL: ("https://www.facebook.com/sharer.php?u="+currentURL, currentURL is the global variable that was assigned earlier and will place whatever the current documents URL is, in place of currentURL. The second parameter, the name: "", This is left blank, as it is optional. The third parameter, the specs: "height=368,width=600,left=100,top=100,menubar=0"); These are a comma-seperated list of items. In my case, I've specified a height, width, and the location of the window, as well as disabled the menubar. Finally, the second statement, return false; tells the browser NOT to follow the link inside the HTML code. If this was not specified, then the browswer would follow the URL in the HTML, AND open a new window. For more information on the window.open method, please see the link at the bottom of this new answer.
fbShare.onclick = function() {
window.open("https://www.facebook.com/sharer.php?u="+currentURL,"","height=368,width=600,left=100,top=100,menubar=0");
return false;
}
gplusShare.onclick = function() {
window.open("https://plus.google.com/share?url="+currentURL,"","height=550,width=525,left=100,top=100,menubar=0");
return false;
}
twitterShare.onclick = function() {
window.open("https://twitter.com/share?url="+currentURL+"&text="+currentTitle,"","height=260,width=500,left=100,top=100,menubar=0");
return false;
}
And finally, I modify the HTML href elements of each social media button so that when the user hovers over the share buttons, they see the correct Share URL displayed in their browsers status bar. The first part of this statement grabs the element id, fbShare and the second part tells it to set an attribute, .setAttribute. Then we pass in the attribute name that we want to change, ("href", in this case, and then we pass in what we would like the new attribute value to be, "http://www.facebook.com/sharer.php?u="+currentURL); currentURL is the same here, as earlier. It is the variable that holds the value for whatever the current page's URL is.
fbShare.setAttribute("href","http://www.facebook.com/sharer.php?u="+currentURL);
gplusShare.setAttribute("href","https://plus.google.com/share?url="+currentURL);
twitterShare.setAttribute("href","https://twitter.com/share?url="+currentURL+"&text="+currentTitle);
That's about all there is to it! I hope I wrote this well and I hope it is relatively easy to follow. If any pros out there have any suggestions, please feel free to toss in and give your advice! :)
My JS file
http://jrltest.host-ed.me/_js/share.js
Link to information on the window.open method at w3schools.com
http://www.w3schools.com/jsref/met_win_open.asp
Link to information on the .setattribute method at w3schools.com
http://www.w3schools.com/jsref/met_element_setattribute.asp
OLD ANSWER: I figured I'd add this as an answer. It does the trick and solves the exact problem that I had. The URL after 'window.open' would be the social media's Share Link (in the case of the example, it's google plus' Share Link. There are a few variables that can be either modified or removed. Anyone that's good with scripting could probably create a PHP version (which I would LOVE) or modify it to better suite their needs. At any rate, I hope this will help someone out!
<a href="javascript:(
function(){
var w=480;var h=380;
var x=Number((window.screen.width-w)/2);
var y=Number((window.screen.height-h)/2);
window.open('https://plus.google.com/share?url='+encodeURIComponent(location.href)+'
&title='+encodeURIComponent(document.title),'','width='+w+',height='+h+',left='+x+',top='+y+',
scrollbars=no');
})();" style="background: url(/wp-content/themes/HTML5/images/googleplus.png) no-repeat scroll left center transparent;">
Share to Google+</a>
Native Window Open function its not a good idea, browsers like Mozilla and Chrome block pop up. I think its better use a plugin to open a new windows with the share url, like jquery popup plugin. Work very fine for me and browser cant block it.
Copy an paste into a new js file like original name: 'jquery.popup.js'
jQuery.fn.popup = function(options) {
var defaults = {
width: screen.width/2,
height: screen.height/2,
titlebar: false,
status: false,
resizable: true,
toolbar: false,
scrollbars: true,
menubar: false
};
var options = jQuery.extend(defaults, options);
Boolean.prototype.setProperty = function() {
if (this == true) { return "yes"; } else { return "no"; }
};
jQuery(this).click( function() {
var target = this.target;
var href = this.href;
var posY = (parseInt(screen.height/2)) - (parseInt(options.height/2));
var posX = (parseInt(screen.width/2)) - (parseInt(options.width/2));
var win = window.open(href, target, 'titlebar=' + options.titlebar.setProperty() + ', screenX='+ posX +', screenY='+ posY +', left='+ posX +', top='+ posY +', status=' + options.status.setProperty() + ', resizable=' + options.resizable.setProperty() + ', toolbar=' + options.toolbar.setProperty() + ', scrollbars=' + options.scrollbars.setProperty() + ', menubar=' + options.menubar.setProperty() + ', width='+ options.width +', height='+ options.height);
win.focus();
return false;
});
return this;
};
USAGE:
<script src="jquery.last.js"></script>
<script src="jquery.popup.js"></script>
<script>
jQuery(function(){
//simple load
jQuery(".popupLink").popup({ width: 640, height: 480 });
});
</script>
<a class='popupLink' href="https://www.facebook.com/share.php?u=<?php echo URL;?>">Share Facebook</a>
ALSO YOU CAN PASS OPTIONS LIKE THE PLUGIN EXAMPLE OPTIONS
<script>
jQuery(".popupLink").popup({ width: 640, height: 480, resizable: false, menubar: true });
</script>
The author website dont exist any more. This are the information that comes with the plugin comments
/*
* jQuery popup v1 - A jQuery popup plugin.
* By Jordan Thomas - http://labs.wondergroup.com
* Licensed under the do whatever you want to license.
* If you like, keep this message intact so
* someone else can find the origin.
*/
You can also use PHP to resize the window but here is the PHP version... Have fun :)
See Also: PHP - Getting Current URL
<?
echo '<a href="https://plus.google.com/share?url='.$_SERVER[HTTP_HOST].$_SERVER[REQUEST_URI].'"
target="_blank"><img src="images/google-custom-icon.png"></a>';
?>
I'm developing a mobile app (iOS 5.0 and above compatible) for a website where users can view certain data directly from their landing page. The user can refine the results by selecting specific options (i.e. location and/or date). They don't offer any web service calls, and won't allow us to access their database. So my only resort of collecting the data is from 'scraping' the site directly.
My issue is that I don't know how make the changes that the users can make on the site from a http request from the phone. For example the site below:
can be pulled from the site and saved as a string with the following command
NSString *html = [NSString stringWithContentsOfURL:urlrequest encoding:NSUTF8StringEncoding error:&err];
I'm able to separate the relevant data:
// I decided to add the script function name in case anyone wanted to look for themselves
<script type="text/javascript" language="Javascript">
//<![CDATA[
function loadData(){
var winMsgTitle = "Date: 04/11/2012";
// this is the actual data I'm concerned with:
gLatLong = new GLatLng(31.59019444444444, -110.50655555555555);
tmpMapIcon = new MapIcon("0_0", 21, 15);
marker = createMarker(gLatLong, 0.0, createInfoWinMsg(winMsgTitle, ".00", ".84", "5711", "2012", "", "07:00"), tmpMapIcon);
map.addOverlay(marker);
point = null;marker = null;
gLatLong = new GLatLng(32.2938260182, -110.7896411419);
tmpMapIcon = new MapIcon("0_0", 21, 15);
marker = createMarker(gLatLong, 0.0, createInfoWinMsg(winMsgTitle, ".00", "1.00", "1254", "2012", "", "07:00"), tmpMapIcon);
map.addOverlay(marker);
point = null;marker = null;
gLatLong = new GLatLng(33.5966853633, -112.1744066477);
tmpMapIcon = new MapIcon("0_0", 21, 15);
marker = createMarker(gLatLong, 0.0, createInfoWinMsg(winMsgTitle, ".00", ".70", "256", "2012", "", "07:00"), tmpMapIcon);
map.addOverlay(marker);
point = null;marker = null;
Note: There are obviously more points, this is just snip it
In order to get another location, or date, I have to manually select via the site and select the options on the right hand pane. My question is, how do I make those changes programmatically via objective-c?
I suppose you use UIWebView to load that page.
One possible solution is to write JavaScript functions to simulate normal user operations on the right hand pane (for example, use jQuery's trigger function to select the drop down list, select 'Date Range', etc.).
These javascript functions could be coded as strings in your app code.
Then, call stringByEvaluatingJavaScriptFromString: on the webview to run these javascript functions:
[webView stringByEvaluatingJavaScriptFromString:javascript]
Here the javascript parameter is a string you construct that calls your javascript functions. For example:
NSString *javascript = [NSString stringWithFormat:#"selectRegion(%#)", targetRegion];
When doing this, I'd suggest you write these javascripts and test them using a browser's debug tool(Safari's Web Inspector, Firefox's FireBug plugin...) to make sure they work as expect first.
If i understand you correctly, you want to pull the data without displaying the website to your user?
If yes, the only solution i can currently think of would be an off-screen UIWebView, in wich you do some JavaScript-magic to simulate user-input, and then get the data out.
This is however a very hackish approach and i would not suggest you use this in a shipping application.
In fact, i would suggest not doing any webscraping at all, because if they change their sites structure, your app fails to work.
If you are really developing an app "for them" then go tell them that you need a webservice in order to write a good application.
If you are not working "for them" but are just trying to write an app that uses their service to get data, then what you are doing is not only complicated, but does most probably also violate their EULA and you should therefore not be doing that at all. Consider contacting them and see if you can get them to work with you in that case.
How can I create test cases according to my requirement.
Example:
I have a form with many fields. There is one field name Father's Name, now I want that the user should insert only string in this field, no numeric values should be accepted.
I wanna carry out such cases and do testing using the tool. How can I do this in TestComplete?
So, you want to validate that the tested application correctly handles the situation when forbidden characters are entered in the field, right? If so, then the exact solution depends on what the application does when a forbidden character is entered:
1) The app shows an error box. In this case, make your test enter a forbidden char and check for the error box existence using the appropriate Wait* method (WaitWindow, WaitNamedChild, etc.). Short example from the top of my head (did not run the code):
var TextToEnter="First 123Name";
EditBox.Keys(TextToEnter);
// As a rule, validationg is performed when the focus changes
EditBox.Keys("[Tab]");
var ErrorBox = MainWnd.WaitNamedChild("wndErrorDlg", 5000);
if (ErrorBox.Exists)
Log.Message("Succeeded - the error box is shown");
else
Log.Error("Failed - no error box detected");
2) The app does not show any error, but just ignores the forbidden chars making them not to appear in the edit box. In this case, just compare the actual text against the expected text. Something like this:
var TextToEnter="First 123Name";
var TextToExpect="First Name";
EditBox.Keys(TextToEnter);
if (EditBox.wText == TextToExpect)
Log.Message("Succeeded");
else
Log.Error("Failed");
I hope this helps.
Let me start off by saying this is my first Flex project, so I've been reading a lot, learning a lot, but am still struggling to make this work.
I am trying to display records from an SQLite database within my Flex mobile application. I have the input form saving data correctly to the database, and I have a datagrid that is outputting the records (so I can confirm everything looks good). However, my problem is making the data readable.
I want the user to be able to select an identifier from either a drop down or a list (even if it's just a 1 column datagrid), and have the labels to the right update to reflect that information.
My thought was to have the datagrid setup as such:
<mx:DataGrid x="10" y="10" width="100" height="500" id="SiteSelector" dataProvider="{siteData}" includeIn="SiteDetails" change="SiteChoice(event)">
<mx:columns>
<mx:DataGridColumn headerText="Site Name" dataField="SiteName" />
</mx:columns>
</mx:DataGrid>
The datagrid shows the Site Name without issue from the database. The change function:
private function SiteChoice(event:ListEvent):void
{
var statement:SQLStatement = new SQLStatement();
statement.sqlConnection = connection;
statement.text = "SELECT * FROM SITES WHERE SITE_ID = ?"
statement.parameters[0] = SiteSelector.selectedItem.SITE_ID;
statement.execute();
siteDetails.source = statement.getResult().data;
}
I also have declared:
[Bindable] private var siteDetails:ArrayCollection = new ArrayCollection();
My question is, how do I get a label to reflect specific column data from the table? I believe I plug it with the siteDetails as the data provider, but how do I specify that I want, say for instance, the address or site contact to appear in a label?
<s:Label text="{siteDetails}" id="ExternalIPLBL"/>
I've been searching Google and the Adobe Developer Connection/livedocs/cookbooks pretty heavily over the last week and still have yet to come up with a working solution.
Try this:
first, you can display a field form an arraycollectiokn like this
siteDetails.getItemAt(0).Street}
The dynamic is the getItemAt(0) Value. Register an clickhandler to the datagrid and debug the event. Somewhere there, the selectedIndex is a part of that event.
By the way, id should start with small letters ->ExternalIPLBL
Set an Breakpoint in the eventhandler, try to find out, where your data are (selectedIndex) and set the label text in the eventhandler:
ExternalIPLBL.text = event.selectedIndex[Street].
BR
Frank