I'm trying to help a client with their slow-ish website https://www.dp-tools.de. If I use Google page speed for mobile I can see that it takes 7 seconds to be interactive but nothing in the found problems really tells me why it is actually that slow. I also tried chrome lighthouse and couldn't really see all that much.
Is there another way of checking or maybe anyone here sees why it is so slow?
Open up the client's Shopify Admin, and examine the theme. Does it have any strange slow code in it? Examine the Apps installed that directly affect theme. Are any of them crap, old, broken junk?
The best way to debug a slow theme is to just start hacking out any junk the client may have added to their theme. A lot of themes are so bad, for example, they load jQuery 3 times. Likely you have one bad apple in there, a call that is blocking and takes way to long to timeout or respond. Developer tools can point these out to in your console. Mobile versions have to come with a developer console you can inspect too right?
Related
I would like to render the following website with Scrapy Splash.
https://m.mobilebet.com/en/sports/football/england-premier-league/
Unfortunately, Splash always gets stuck at the loading screen:
I have already tried using a long waiting time (up to 60 seconds) with no results. My Splash version is 3.3.1 and obey robots.txt has been set to false.
Thanks!
There's not quite enough info to answer, but I've got a good guess.
You see, the major difference between Splash and your browser is the user agent string. You have one that looks like a person. Splash generally doesn't.
This kind of infinite loading is a method used by sites to mitigate repetitive load. Often when you're developing locally without a proxy you'll trip these issues. They are quite maddening to develop against because they're inconsistent.
Your requests are just getting dropped, you'll probably see a 403 after 5-10 minutes.
I think it's likely you can solve this issue with the method mentioned in this answer: Scrapy+Splash return 403 for any site.
I don't think it'll be possible - this website needs JS to be rendered. So you'll need to use something like Selenium to scrape information from it.
Also, perhaps what you are looking for is an API for that information - since scraping it from a website can be very inefficient. Try googling "sports REST API" - look for one with Python SDK.
Ok, so Splash is supposed to render the JS for you it seems. But I wouldn't rely on it too much - those websites constantly change and they are developed against latest browsers, your best bet is to use Selenium with Chromium driver (though using API is much more preferable).
Many of our users are reporting that they are getting a blank page when using IE11 to access our website. Sometimes they don't even get a blank page, the browser just stays on the last page visited. These users can access other domains (such as google.com) without a problem.
For the browsers that are failing, if those users disable Protected Mode in IE, then they can again access our site, but this isn't a good solution because we have thousands of users and we can't go telling each of them to reconfigure their browsers, not to mention that we're completely losing the business of those potential customers who just surf by, see a blank page, and then keep on going without filing an error report.
Firefox and Chrome work fine. Also, even when using IE11 in protected mode, some users have zero problem, their computers just seem to work.
We are running the site off of IIS7. Other sites on the same server run fine, it's just the one particular site that is having the problem, and it's intermittent, affecting some computers and not others.
There must be something I can do on my side in the server or network settings to keep this problem from happening, but I'm baffled as to what it might be. When I look at the network traffic on a browser that's failing, the GET request is just being aborted with no explanation. When looking at the traffic with Wireshark, I see no errors, and no errors are showing up in the IIS logs. Of the browsers that fail, they are not even opening a connection to our web server to make a GET request, the request is just aborting immediately.
Any advice appreciated!
(followup): Another test we did: We can reproduce the problem on our development server, with the same pattern of which computers "work" and which don't. We tried turning off the webserver, and the problem stayed consistent, we'd still get a blank page error. So it's obviously not to do with anything that's in our page content? I've run tests on 9 computers on our office network: 5 worked, 4 failed. We're all baffled. :/
(January 24 followup): We figured out what's causing the problem, but not how to fix yet. Deep in the Windows registry, a key is being set in a Zonemap folder, adding a "play.net" domain with a key value of 2. IE sees that key when someone types play.net, and quivers and dies without an error message (Firefox and Chrome handle it fine).
So next question, what is setting that key in the first place? Probably an ActiveX control somewhere, but we haven't had any luck finding it in this 15 year old site, as many of the coders who may have created ActiveX controls in the past are long gone.
Does anyone know of a way to scan an entire domain and list anything that might be twiddling a registry key?
Followup mail threads from Elonka indicated that some users had the play.net domain mapped to a specific / non-default Internet Explorer security zone, and those users were the ones having problems.
Try add
<meta http-equiv="X-UA-Compatible" content="IE=10" />
into the header. It fixed IE11 for my website. It just forces it into IE10 mode (A complete joke though)
I accessed your domain in Internet Explorer 11, with Protected Mode enabled. I was unable to get a fully broken experience, but I did see something perhaps related to what your customers are experiencing. Some directories on your domain would hang for me, resulting in a white screen if they've not yet been loaded, or staying on the present page while the forthcoming page is loaded.
When I navigated to /dr, I found that my browser hung for over 7 seconds while your files loaded. Note, there were roughly 60 items, totaling about .78mb. This isn't much, but I'm presently on a relatively slow connection, and this resulted in a hang for me.
I would inquire what connection speeds your clients are on when experiencing this issue - it could very well be nothing more than some directories being served up more slowly than others. Regardless, you could optimize the experience considerably by compressing and concatenating your CSS files, as well as all of your JavaScript files. This results in fewer parallel requests, and quicker downloads.
If you have a reproducible example, please feel free to share and I'm sure myself and many others here will be more than happy to assist you.
For me box-shadow on an element caused this behavior. I removed the box-shadow(but left -webkit-box-shadow and -moz-box-shadow) and the problem disappeared. The interesting thing is that I still have box-shadow on another element...
If you are using 'offline.js' in your project, and you have version less than 0.7.19, then IE11 latest security update will see this code as potential security threat and block pages which are using 'offline.js' code.
Solution: Update to latest 'offline.js' version.
Is there any way to consistently detect PhantomJS/CasperJS? I've been dealing with a spat of malicious spambots built with it and have been able to mostly block them based on certain behaviours, but I'm curious if there's a rock-solid way to know if CasperJS is in use, as dealing with constant adaptations gets slightly annoying.
I don't believe in using Captchas. They are a negative user experience and ReCaptcha has never worked to block spam on my MediaWiki installations. As our site has no user registrations (anonymous discussion board), we'd need to have a Captcha entry for every post. We get several thousand legitimate posts a day and a Captcha would see that number divebomb.
I very much share your take on CAPTCHA. I'll list what I have been able to detect so far, for my own detection script, with similar goals. It's only partial, as they are many more headless browsers.
Fairly safe to use exposed window properties to detect/assume those particular headless browser:
window._phantom (or window.callPhantom) //phantomjs
window.__phantomas //PhantomJS-based web perf metrics + monitoring tool
window.Buffer //nodejs
window.emit //couchjs
window.spawn //rhino
The above is gathered from jslint doc and testing with phantom js.
Browser automation drivers (used by BrowserStack or other web capture services for snapshot):
window.webdriver //selenium
window.domAutomation (or window.domAutomationController) //chromium based automation driver
The properties are not always exposed and I am looking into other more robust ways to detect such bots, which I'll probably release as full blown script when done. But that mainly answers your question.
Here is another fairly sound method to detect JS capable headless browsers more broadly:
if (window.outerWidth === 0 && window.outerHeight === 0){ //headless browser }
This should work well because the properties are 0 by default even if a virtual viewport size is set by headless browsers, and by default it can't report a size of a browser window that doesn't exist. In particular, Phantom JS doesn't support outerWith or outerHeight.
ADDENDUM: There is however a Chrome/Blink bug with outer/innerDimensions. Chromium does not report those dimensions when a page loads in a hidden tab, such as when restored from previous session. Safari doesn't seem to have that issue..
Update: Turns out iOS Safari 8+ has a bug with outerWidth & outerHeight at 0, and a Sailfish webview can too. So while it's a signal, it can't be used alone without being mindful of these bugs. Hence, warning: Please don't use this raw snippet unless you really know what you are doing.
PS: If you know of other headless browser properties not listed here, please share in comments.
There is no rock-solid way: PhantomJS, and Selenium, are just software being used to control browser software, instead of a user controlling it.
With PhantomJS 1.x, in particular, I believe there is some JavaScript you can use to crash the browser that exploits a bug in the version of WebKit being used (it is equivalent to Chrome 13, so very few genuine users should be affected). (I remember this being mentioned on the Phantom mailing list a few months back, but I don't know if the exact JS to use was described.) More generally you could use a combination of user-agent matching up with feature detection. E.g. if a browser claims to be "Chrome 23" but does not have a feature that Chrome 23 has (and that Chrome 13 did not have), then get suspicious.
As a user, I hate CAPTCHAs too. But they are quite effective in that they increase the cost for the spammer: he has to write more software or hire humans to read them. (That is why I think easy CAPTCHAs are good enough: the ones that annoy users are those where you have no idea what it says and have to keep pressing reload to get something you recognize.)
One approach (which I believe Google uses) is to show the CAPTCHA conditionally. E.g. users who are logged-in never get shown it. Users who have already done one post this session are not shown it again. Users from IP addresses in a whitelist (which could be built from previous legitimate posts) are not shown them. Or conversely just show them to users from a blacklist of IP ranges.
I know none of those approaches are perfect, sorry.
You could detect phantom on the client-side by checking window.callPhantom property. The minimal script is on the client side is:
var isPhantom = !!window.callPhantom;
Here is a gist with proof of concept that this works.
A spammer could try to delete this property with page.evaluate and then it depends on who is faster. After you tried the detection you do a reload with the post form and a CAPTCHA or not depending on your detection result.
The problem is that you incur a redirect that might annoy your users. This will be necessary with every detection technique on the client. Which can be subverted and changed with onResourceRequested.
Generally, I don't think that this is possible, because you can only detect on the client and send the result to the server. Adding the CAPTCHA combined with the detection step with only one page load does not really add anything as it could be removed just as easily with phantomjs/casperjs. Defense based on user agent also doesn't make sense since it can be easily changed in phantomjs/casperjs.
ive written a script that configures a bunch of items and displays it on screen using javascript. Now we are in the process of trying to take a screenshot of it via a browser. This particular webpage is for use by the general public, and once the screenshot is taken, we post the photo to facebook for the user.
Ive trialled a number of services that take screenshots but they are all pretty slow, mostly averaging around 25 - 25 seconds to take the shot and have it saved on the server. All our users would leave in this time =(
Ive also tried the application from http://www.websitescreenshots.com/ and whilst its brilliant i cant get it to render our page with the js on it.
I have found this to be a common thing amongst screenshot software, and ive even gone through to trying to take advantage of the free webservice from Wordpress MShots.
Whilst it took photos nicely you could never tell how long it was going to take and not good enough for a production website.
So can anyone recommend a utility that will allow screenshots of URL's accessible via a command line that can render JS and also runs on windows server 2008 ? I am having no luck tryning to source an app that does this..
I am trying to convert one of my earlier written time tracking apps to a web browser based app.
Some options of technology include java, flash, ... My goal here is letting a user track time for our organization while having ideally no extra installation on desktop, or least minimal, like flashplayer or jvm, etc, but the product itself should be in browser.
I have done some research and so far i can't find a way any technology, of the ones i have known, could take a screenshot of whole desktop while running inside web browser even if, say, browser is out of focus.
The best i could do was take a screenshot of the page on which that app will run. Any ideas? Possible? leads?