I am working on a web scraping project. do any body have idea of scraping dynamic content.
Dynamic content on base of query string is similar to static content but dynamic content based on some event of a control within same page is the point where i am stuck. because in this case page url remain same.
I am using C#.
Thanks in advance
Your question is rather general.
I'm not sure what you mean by event of a control, but as long as a browser generates http request you can catch it using tools like Firebug for Firefox or tools built in Google Chrome and see what is actually being sent to the server. So called AJAX requests are nothing else than standard http requests, it's just that web page is not reloaded as a whole.
Based on that information and page source it is possible to figure out how to create range of reguests that would simulate user interaction with dynamic elements on the page.
Related
I'm currently wondering about the difference between static and dynamic pages when using vue.js.
Are vue.js pages static or dynamic?
They can change on user interactions but not necessarily need to interact with the server to change data, when not using axios or alike. So does it still count as static just changing the frontend itself on input or interactions?
Little confused about this. Hope for good explanations.
A static page is a page delivered to the user exactly as stored and with no chance on being changed, end of story!
Although dynamic pages can take 2 types: Client-side dynamic web page and a Server-side dynamic web page.
Client-side dynamic page is changed without server requests. For example, when you click a button and something pops on the screen, or some content on the page changes, etc... (i think you take the idea).
Server-side dynamic page is changed with server requests, as you said for example, with HTTP requests using Axios.
And this is not the Vue.js definition, is the default definition for static and dynamic pages.
I'm trying to extract all the article text from the following site:
https://www.phonearena.com/reviews/Samsung-Galaxy-S9-Plus-Review_id4494
I tried findAll(text=True) but it extracts lot of useless information.
So I did findAll(text=True, recursive=False) but it ignores text data in certain tags like ? What's the most effective way of extracting the text in this case?
The website seems to be javascript protected. It loads the body content when requests already retrieved the http response. You need to simulate a real page request. With the python module Selenium Webdriver it would be possible.
I am performing an integration to a third party site and they have asked me to redirect to their URL with a bunch of POST variables.
The only way I can work out how to do this is by creating an HTML form with hidden fields, and then trigger a JS click event on a button to POST the form to their site.
Ideally I would like to do this from the VB code as the data may be sensitive and there are bits I don't want to render in the client side.
Has anyone any experience of doing this successfully. I have googled around but can only find ways to use GET variables on the URL or POST and read the response.
I have a task to do where I need to make calls to an external xml api to fetch data for an event calendar in the sidebar of a site. The date will be changed with JavaScript and then i need to make another call to refresh the data. Can somebody give me an idea about how to cleanly set up an action or function somewhere that I can direct an Ajax action to? It's easy to set up a widget with the correct HTML etc but where does the Ajax connect to? Ideally when the content initially loads on the page it would use the same function that the post is going to use to generate the HTML on the server side.
Any tips would be appreciated. This is an xml api...no option for json or jsonp so credentials including a token and user I'm assuming will have to go somewhere in my widget, something like a proxy function?
It sounds as if you're asking about making cross-domain AJAX requests, AKA the "same origin policy."
The same origin policy prevents document or script loaded from one origin from getting or setting properties of a document from a different origin (domain). See http://www.mozilla.org/projects/security/components/same-origin.html for a more detailed description of the policy.
See Ways to circumvent the same-origin policy for a good description of the options available to circumvent this limitation.
The short answer is that unless you have control over the domain to which you're making AJAX requests, your best bet is probably to set up a simple proxy that lives in the same domain where your AJAX is running, which will forward requests to the destination. Doing a google search on "simple AJAX proxy" will get you a host of results, including pre-built proxies in a variety of languages.
I am trying to set a web application using ajax dojo technology. For this I need an exemple showing how to send and receive data with dojo library.I use also J2ee in my project.
Please if you have a constructive example show it to me.
Client side: JSP with ajax dojo its purpose is to send a selected item to the server side.
Server side:Servlet catch the data sent and construct sql query with it finally send it again to jsp
JSP receive the result of the query (xml file)
dojo construct the seceond dropdownlist automaticallt.
Just i need an exemple of send and receive.
Many Thinks.
Have you looked on Dojo's website? From your question, it sounds like what you want is dojo.data.ItemFileReadStore. The Dojo website has many examples of the uses of ItemFileReadStore, including one in which one FilteringSelect is dependently linked to another, which seems to be what you describe.
As to your request for examples of send and receive in JSP: a Google search for JSP tutorial turns up a site called jsptut.com as the first result. Have you looked there?