I've seen examples such as this one showing how to login to facebook with casperJS:
How to login into a website with CasperJS?
but am unable to get this code to work for me. I'm not interested in the login portion, I just want to load any facebook page into casperjs or phantomjs but keep getting a fail on load.
Is this working for anyone else? or has facebook detected the browser and not allowing access anymore?
Here is a simplified version of what I am unable to do:
var casper = require('casper').create({
verbose: true,
logLevel: 'debug',
pageSettings:{
userAgent :'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.94 Safari/537.4' ,
loadImages: false, // The WebPage instance used by Casper will
loadPlugins: false // use these settings
}
});
casper.start()
casper.thenOpen('https://www.facebook.com/pfchangs', function() {
this.echo(this.getHTML());
//this just prints out empty page: <html><head></head><body></body></html>
});
casper.run();
I'm running this on Windows 7
Looks like this is known bug with casperjs windows batch file:
https://github.com/n1k0/casperjs/commit/0d659f140f1e2120bed967d8301657b5fe79f19c
Related
My question is related to:
Scrapy-splash not rendering dynamic content from a certain react-driven site
But disabling private mode did not fix the issue (passing --disable-private-mode as an argument to the Docker container).
I made a single page app using React and all it does is add an element to root:
function F(){
class HelloMessage extends React.Component {
render() {
return React.createElement(
"div",
null,
"Hello ",
this.props.name
);
}
}
return(React.createElement(HelloMessage, { name: "Taylor" }));
}
But Splash only shows a blank white page!
I also made a single page website using Django and the HTML loads a script on head:
<script src="/static/js/modify-the-dom.js"></script>
And all the script does is to write something on the page after the content is loaded:
window.onload = function(){
document.write('Modified the DOM!');
}
This change shows up on the browsers! But on on Splash, if I remove the window.onload and just write document.write('Modified the DOM!'); it works fine! So the issue seems to be with window.onload even though I'm waiting long enough about 10 seconds on the LUA script.
So I was wondering how to fix that.
I also checked my Splash with this website and it seems JS is not enabled!
https://www.whatismybrowser.com/detect/is-javascript-enabled I wonder if there's an option I have not enabled?
One example of a client-rendering website that Splash does not crawl properly (same issues above) is https://www.digikala.com/ it only shows a blank white page with some HTML that needs to be populated with AJAX requests. Even https://web.archive.org/ does not crawl this website's pages properly and shows a blank page.
Thanks a lot!
Update:
The Splash script is:
function main(splash, args)
splash:set_user_agent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36")
assert(splash:go(args.url))
assert(splash:wait(30)) -- Tried different timeouts.
return {
html = splash:html(),
png = splash:png(),
har = splash:har(),
}
end
I also tried setting these to no avail:
splash.plugins_enabled = true
splash.resource_timeout = 0
I am working on a React Native app that displays HTML content using the react-native-render-html library. This HTML data is coming from content formatted in WordPress. The same content is also being displayed in the web app made with ReactJS. I am also using this iframe plugin, as recommended by the render-html library, to display the iframes in my HTML.
I have the following problem with rendering Twitch video embeds.
The Twitch embeds are in this format:
<iframe src="https://clips.twitch.tv/embed?clip=ShortHelpfulSquirrelCmonBruh-hWj49qxBfx-VKseO&autoplay=false&parent=my.parent.domain" width="640px" height="360px" frameborder="0" scrolling="no" allowfullscreen="true"></iframe>
Please note that these are not live streams, but clips. The ReactJS website displays these embeds correctly with the thumbnail:
With the same embed code, however, my React Native app displays the embed without the thumbnail but with just a black background (although it shows the title and controls):
Once I play and pause the video, the background does not remain black, however:
Here is the rendererProp that I am passing to the library for iframes:
iframe: {
scalesPageToFit: true,
webViewProps: {
scrollEnabled: false,
mediaPlaybackRequiresUserAction: true,
javaScriptEnabled: true,
domStorageEnabled: true,
userAgent: Platform.select({
ios: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.2 Safari/605.1.15",
android: "Chrome/56.0.0.0 Mobile android",
}),
},
}
I am passing the IframeRenderer and iframeModel as imported directly from #native-html/iframe-plugin (linked at the beginning) into the renderers and customHTMLElementModels props of the <RenderHtml/> component. Also I am using the WebView from react-native-webview lib.
My team and I cannot figure out why the thumbnail is not appearing in the react-native-webview, but works fine in the website. If anyone has any experience rendering Twitch video iframe embeds on react-native-webview, please take a look. Any help will be appreciated.
I am having difficulty debugging why Splash is unable to load the javascript properly for this website: https://www.bigc.co.th/2000003703081.html
I have referred to this answer to disable Splash from running in private mode and also other solutions here in the docs, such as increasing wait time and setting user agent.
However, the website is still loading as though without JS on Splash, as seen in this picture below
I am currently debugging why the website is not loading properly for the purpose of incorporating this with scrapy-splash code.
Here is my current script. Thank you for your time and assistance in advance!
function main(splash, args)
splash:set_user_agent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36')
splash.private_mode_enabled = false
splash:set_viewport_full()
splash:go{url=args.url}
assert(splash:wait(1))
return {
html = splash:html(),
png = splash:png(),
har = splash:har(),
}
end
I am using the Web SDK for Agora.io and trying to implement video chat feature and have screen sharing enabled. However, I am getting this issue:
MediaError "Access to screen denied
Seems like your web page does not have the right to access the device. Not sure which browser you are working with. If your browser is Chrome/Chromium, try: Change site permissions.
Or test your browser with their tools: Agora Web Trouble shooting, Agora Web Demo
You can use Chrome or Firefox with Agora.io to screenshare. Make sure that you are deploying to HTTPS.
Firefox:
screenStream = AgoraRTC.createStream({
streamID: uid,
audio: false,
video: false,
screen: true,
mediaSource: 'screen' // 'screen', 'application', 'window'
});
Chrome:
screenStream = AgoraRTC.createStream({
streamID: uid,
audio: false,
video: false,
screen: true,
//chrome extension ID
extensionId: 'EXTENTION-ID-HERE'
});
Are you using Chrome? Check a few things:
Check that the extension ID matches
Check the domain name matches in the manifest.json file
Chrome Plugin Reference page
I am using CasperJS 1.1.0 with PhantomJs 2.0. I am using Linux server.
I have a script, trying to run and take screenshots.
This screenshot is fine for local server nor for live.
Screenshot of the site will look like this:
https://i.stack.imgur.com/dJe4L.png
So, instead of the actual letters, it renders square boxes.
var casper = require('casper').create({
verbose: true,
});
casper.options.waitTimeout = 40000;
casper.options.viewportSize = {width: 1200, height: 768};
phantom.cookiesEnabled = true;
casper.start('http://domain.com/', function() {
this.wait(5000, function () {
this.echo(this.getTitle());
this.capture("screenshot.png");
});
});
casper.run(function () {
this.exit(0);
});
Please tell me where i am wrong.
I would use waitForSelector on the items that you're wanting screenshots of, rather than the 5 second hard wait. Consider adding
verbose: true
waitTimeout:20000
Which will give you 20 second wait limit, longer than the 5 seconds default. Then, instead of your current wait, use something like:
this.waitForSelector('img', function() {
this.capture('test.png')
}
You can change the img selector to be something more specific, but waiting for the images themselves to load is a pretty certain way of making sure they will be captured. The reason we changed the timeout to 20 seconds is in case their site takes a little longer to load their images :)
Let me know if this doesn't help, happy to follow up. I will need examples of the site and the image you'd like though.
EDIT: Can confirm this works for me, let me know if it works for you.
var casper = require('casper').create({
verbose: true,
logLevel: 'error',
pageSettings: {
loadImages: true,
loadPlugins: true,
userAgent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11'
},
clientScripts: ['vendor/jquery.min.js', 'vendor/lodash.js'],
viewportSize: {
width: 1600,
height:1000
},
});
casper.start('http://dev.bunkerbradleycouture.com/')
casper.then(function() {
this.waitForSelector('img', function() {
this.capture('test.png')
})
})
casper.run()