I am trying to scrape the new.google.com and get some information, I have no issue locally but when I deploy to our Datacenter it fails with "connection refused" that means it is blocked.
Get https://news.google.com?ceid=en%3Agb&gl=en-gb&hl=en-gb&hs=en-gb&pz=1: dial tcp 172.217.5.206:443: connect: connection refused
Do we have any alternative of passing header and by-pass to get unblocked? or is using paid API is the only option if I have to use google for some of my testing?
Did anyone encounter and resolved it?
This is pretty common. Somebody else used the same IP address for scraping or even something worse :) so it's blocked.
You can use some proxy services. There are some with free tier so it will do the job for testing.
And before you ask... free proxy services are super slow and probably already blocked :)
Related
Reason behind error524 connection timeout.
How to fix it even though the website havent cover high storage
Server origin that is being used is from India.
website :https://motogenes.com/
trying to fix the error.
pause the running website then again enable to run the website via cloudflare
.
Error 524 is usually related to a connection timeout issue between Cloudflare and the origin server. This means that Cloudflare was unable to establish a connection with the origin server in a timely manner.
It's hard to tell what is the issue without debugging the server, but I would start by checking the firewall and network settings.
In documentation of Cloudflare, they say that you can change value of timeOut but only accounts Entreprice
https://api.cloudflare.com/#zone-settings-get-proxy-read-timeout-setting
How is the discord API detecting your IP when rate-limiting your computer?
Even by doing requests trough tor and resetting the connection every 5 requests to change my IP, it still rates limit me (you probably know what I am doing, just note that it's for fun, quarantine is boring)
How does it know it still your computer? How does it work?
Exposing an IP is a fundamental part of how the internet works. When you connect to a service, you are sending data to its IP address, including your IP address so that the service may reply to you. There's no way around this, as if the IP given was incorrect, you would not get a reply from that service. Changing your IP using a proxy, VPN, or like you've been using, TOR, is still exposing the IP address of the end point of the proxy, so that the service can respond to the proxy and have the proxy send the request back to you.
Typically, if you are hitting rate-limits that often, you are doing something which is not permitted by the service you are using. If you continually hit rate-limits, the service will catch on and apply harsher rate limits, or even terminate your account. In discord especially, hitting rate-limits that often would indicate you are performing requests with malicious intent. If that's not true, you should re-evaluate how you're going about what you're doing, as there will be a better solution to your problem.
I've been trying to create a Google Compute Engine network load balancing health check for an HTTPS (port 443) endpoint. The same endpoint when accessed over HTTP (port 80) is healthy. Also, the HTTPS endpoint, when accessed for example with curl correctly returns a 200 OK response, which would be the required condition for an healthy check.
It would be extremely helpful if there was a way to access a more detailed error report of why the health check is failing, because it's probably something quite easy to fix, but the total lack of detailed information in the web interface makes it random guess work. Trying to research information about where to find detailed information about why a health check is failing I have come up empty.
I believe this is because load balancer health checks don't currently support HTTPS.
The 200 ok is related to the health check, but if your TCP connection is not closed properly, this can cause this issue. If you run this command: tcpdump -A -n host your_host ip, you can confirm that the TCP connection is closed with a FIN/ACK status. If you see [R] flag in the output, it indicates that the connection is being reset instead of closing properly.
For more information, visit this link https://developers.google.com/compute/docs/load-balancing/health-checks#steps_to_set_up_health_checks
I need answer to one very basic question.
Is it necessary to have telnet access to hit an API while systems are connected over VPN? For example, if my system exposed an API for other systems to hit and they are connected in VPN using IPsec, does a third party system needs telnet access to my server for using that API? The API uses soap protocol for receiving request and sending response.
(I did not find out the solution using google. The question is so naive that I had to assume everyone must already know the answer and does not bother to discuss it in web. Sorry for bothering with this simple question)
This is very strange. Accessing an http endpoint for anything else than dev using telnet doesn't really make much sense to me. If someone is using telnet to fetch informations from a server in an application. Something is already really wrong. If telnet is timing out while doing http requests. It's not really your fault and you shouldn't have to worry about edge cases like this.
If the dev is using telnet to discover security issues. This is a different issue and you could probably log anything that come from this particular client. If you gave hime some credentials, it should be easy to find who is doing which request. (I believe you might be already doing this).
You should probably ask the dev "why are you using telnet?". If for whatever reason the dev though he could send a plain SOAP request to your server using telnet without sending HTTP headers and so on.... then yes the connection is likely to timeout because the server isn't going to handle the request.
In my twisted mind, I can imagine some kind of legacy application calling scripts that open telnet sessions to parse some data and return the "parsed" data to the patched legacy project that doesn't handle http/tcp. I'd have in mind some sort of old Cobol application. Much more easier to do system call than to rewrite the whole thing to support APIs.
If for whatever reason, the client claims that telnet is needed for whatever reason. You can tell him back that telnet shouldn't be considered secure. Your api can be accessed using https. As far as I remember telnet doesn't encrypt anything unless you send encrypted data. If your client was able to hack a solution using telnet, I'm pretty sure they can hack a different solution wich use an actual http client.
Well I decided to try make a proxy checker, like Charion or Elite proxy checker. These programs accept large lists of proxies in the IP:PORT format, ping them tell you the response time
see the screenshot of Elite Proxy Checker, im trying to make a simplified version of this program. http://i52.tinypic.com/a57slh.jpg
I investigated and made my checker using Ping.SendAsync(ip, timeout, ip) method.
It was only afterwards that I discovered that you can only Ping IP's using this method, not the ports as well.
Ive spent a few hours trying to find the correct class/methods in order to be able to ping ports, reading different forum posts from experts they say its impossible to ping ports only IPS, can only use sockets to try open a connection with the port.
However, I have seen programs that people have coded in VB.NET that ping in the IP:PORT format, ie lets you choose timeout, tells timeout, etc.
My question is , what classes methods should I be using to do this ?
Im pretty sure its not sockets... theyve got to be pinging the individual ports as well somehow.
Any help would be appreciated.
Cheers,
(I code for a hobby, im not a pro, so sorry if I make glaring errors)
Have a look at tcpping, here is the Windows version:
http://www.elifulkerson.com/projects/tcping.php