I have been using proxy ips with phantomjs to scrape data. Has someone used luminati.io with phantomjs? since luminati uses end user computer ips to read pages. Its costly and I need to know if someone has used this already and I should try this.
Thanks,
You need to pass --proxy-auth and --proxy to the command line, for example:
phantomjs --proxy-auth=lum-customer-YOUR_CUSTOMER-zone-YOUR_ZONE-country-YOUR_COUNTRY --proxy=zproxy.luminati.io:22225 script.js
Related
I am trying to set up the access control for my personal website on synology NAS. Right now I am using DS212J. I found the following article teaching how to use "htaccess" on apache to achieve that.
http://www.synology.com/support/faq_show.php?lang=enu&q_id=347
But I encountered 2 problem:
1) I don't have the command "htpasswd" after ssh to my DS212J.
2) After I manually create those files, I got the username and password prompting up on the website. However, the password in "admin.pw" and "normal.pw" doesn't work when I type in. I am not sure whether it is because I didn't use command "htpasswd".
What am I missing here?
Thanks.
The htpasswd executable is at /usr/syno/apache/bin/htpasswd. As that directory is not on the PATH, you have to explicitly use the whole path to use it:
Apache doesn't keep passwords in plain text, it uses a hash of them so that even if someone gets access to the password file they won't be able to log in. That's why you must use htpasswd to create the file.
For anyone who comes across this, I couldn't get that synology support tutorial to work either. I am running DSM 5.1-5022 Update 4 on my own domain with ssl. Here's how it got it working:
Follow the steps in the following wiki: Synology Wiki
In the control panel, go to "Web Services" and disable SPDY for secure connections. (Gleaned from this article.)
Good luck.
Is there any way to capture post request and write it to log running apache mod_proxy (or any other mod)?
For example, I have one CMS behind apache mod_proxy and I want to capture Login textbox which uses POST verb in the apache log file, it is possible?
Thanks :).
Please take a look at mod_dumpio. All input and/or all output will be logged into error.log.
mod_security can log post data too, but a little complex.
This may suit to your needs mod_log_post (striped down version of mod_sec), but has less documentation and support. Though it might work within your purpose.
I am trying to load-test a web application using JMeter. I used proxy and then run the test. I am able to login into the application, but for some pages it is giving response code 4** in JMeter.
After some homework I found that the request I am sending through the browser and Jmeter might be different, so to check the same I am using fiddler. Is this the correct approach to find the root cause of the problem? What all things/attributes I should compare in fiddler and Jmeter for such requests?
Any help is appreciated. If you want, I can provide more information.
Thanks a lot :)
Abhijeet,
Edit the jmeter.properties file which is located in the bin directory.
Un-comment the proxy.content_type_include=text/html|text/plain|text/xml
Save the file and restart Jmeter.
Try recording and see if this makes a difference.
Is there a way to detect if a site is on a Content Delivery Network and if yes, can we tell which service are they using?
A method that is achievable from the command line is using the 'host' command, with the -a flag set to see the DNS record e.g.
host -a www.visitbritain.com
Returns:
www.visitbritain.com. 0 IN CNAME d18sjq5nyxcof4.cloudfront.net.
Here you can see that the CNAME entry tells us that the site is using cloudfront as the CDN.
Just take a look at the urls of the images (and other media) of the site.
Reverse lookup IP's of the hostnames you see there and you will see who own them.
I built this little tool to identify the CDN used by a site or a domain, feel free to try it.
The URL: http://www.whatsmycdn.com/
You might also be able to tell from the HTTP headers of the media if the URL doesn't give it away. For example, media served by SimpleCDN has Server: SimpleCDN 5.6a4 in its headers.
cdn planet now have their cdn finder tool on github
http://www.cdnplanet.com/blog/better-cdn-finder/ The tool installs on the command line and allows you the feed in host names and check if they use a CDN.
If Website using GCP CDN you simply check it using curl
curl -I <https://site url>
In reponse you can find following headers there available
x-goog-metageneration: 2
x-goog-stored-content-encoding: identity
x-goog-stored-content-length: 17393
x-goog-meta-object-id: 11602
x-goog-meta-source-id: 013dea516b21eedfd422a05b96e2c3e4
x-goog-meta-file-hash: cf3690283997e18819b224c6c094f26c
Yes you can find by
host -a www.website.com
Apart from some excellent answers already posted here which include some direct methods which may or may not work for all the websites out there, there is also an indirect way to see if a CDN is there. And especially if its your own website and you want to know if you are getting what you are paying for !
The promise of a CDN is that connections from your users are terminated closer to them so that they get less TCP / TLS connection establishment overhead and static content is cached closet to them so that it loads faster, puts less strain on your origin servers.
To verify this, you can take measurements of site load times across the globe and see if all the users get similar loads times. No you dont have to get a machine everywhere in the world to do that ! Someone has already done that for you
Head to https://prober.tech/ and the URL you wish to test for load times.
Because this site itself is in Cloudflare's CDN, you can put that link itself in the test box and use it as baseline !
More information on using the tool can be found here
I'm not sure if this is possible or not.
What I'm looking for is a way to tell telnet to use a certain IP address to log into and then run commands where the commands change based on a user's MAC address.
Basically it would be:
tell telnet to use x.x.x.x as the IP to log into and put in the correct username and password
tell telnet to run commands (based on the user's MAC address) that can change based on which user stats you want to see, for example: show macaddress
export the output to notepad
close
expect can do this. If you don't have Tcl but Python, try Pexpect.
If you just want to run one command, use ssh (which allows you to log in, run a command and which will return with the error code of the command, so you can handle errors, too).
If you want to run more than a single command, write a script, use scp to copy that script to the other side and then execute the script with ssh. I've used this approach with great success to build a simple spider that could run a script to gather system information over a large number of hosts.
I think you're looking for expect (it automates these kind of interactive applications). Here is a gratis chapter from the authority on expect, the book "Exploring Expect".
Also you should use SSH if this is over the internet. Telnet is insecure as it's a plain text protocol.
Not to blow my own horn, but you may be able to twist a personal app of mine (note: Sorry, I've removed this.) to this end.
There's currently no documentation other than what is on that page and no public source code (though I've been meaning to get onto that, and will work that out tomorrow if you're interested), but I'd be happy to answer any questions.
That said, any MUD client could be turned to the same use too.