I use Postman to test an API (The API of brickset.com). Whe calling my https://brickset.com/api/v3.asmx/login?apiKey=xxx&username=xxx&password=xxx I get the HTML page of cloudflare captcha verification. When doing the same GET in my browser (Edge) I had the captcha page the first time then now I get the correct result.
How can I whitelist my IP or whitelist my Postman app or get the cookie or whatever I need to make sure that when I send my request I bypass this robot verification page?
New information
After testing Postman again with my VPN connected to Hong Kong first, then Belgium.
On Hong Kong, where I live, the request response is this captcha page.
On Belgium, with VPN, the request response with correct JSON.
What is this? How Can I tell this cloudflare system they can trust my IP ? I'm lucky I have a VPN.
If you own brickset.com, you can definitely whitelist your IP through Cloudflare dashboard; brickset.com > Security > WAF -> Tools, and then add your IP.
Since you mentioned it's an API, you probably want to disable Browser Integrity Check so you don't get captcha on API calls for normal browser calls as well (read more about it here).
Can't say how Cloudflare's captcha or anti-bot works. Multiple factors such as public IP reputation, number of requests, User Agent, ...etc, could trigger the captcha page.
Related
I am attempting to build a webapp using Vue for the frontend and Flask for the backend that reads in the users Gmail emails.
Desired functionality:
User clicks a button to "Link Gmail Account" on the frontend
User is authenticated with gmail Oauth2 and confirms. Once confirmed, they redirect back to the page they were on
Once the user confirms, the backend queries gmail to get all of the users emails and returns the data to the frontend
I have been trying to use https://developers.google.com/gmail/api/quickstart/python as a starting point, but I cannot authenticate the user -- I keep getting a redirect uri mistmatch error with a random port (I am doing this locally so have set the redirect uri to be the localhost port where I access my project).
I think I am doing something fundamentally wrong or not using the Gmail API in the correct way, but have searched all over google and youtube to no avail.
Specific things that I think could be causing an issue:
What is the best overall strategy to implement this? Should I use the Gmail API in Python or Javascript? Right now, the use clicks the "Link Account" button which calls an API in my backend which then runs the code in the Python Quickstart guide.
What kind of google project should I set up? I currently have my credentials configured for a "web application"
What should I put as the redirect uri? I am using localhost but am unsure exactly what to put here (I have tried http://localhost, http://localhost:5000, http://localhost:5000/, http://localhost:5000/emails [this is the url I want them to return to]). No matter what I put, I keep getting a redirect uri mismatch and it says the uri it is looking for is http://localhost:[random port]/
I would appreciate any help on how to approach achieving this. Thank you!
Depending on what you are going to use Gmail API for, you must select the device or category. In your case, as it is a website it should be set to "Web Application".
Also, you should be using the following redirect URI: http://localhost/emails/. You should not include the port number and you should be using trailing slashes (adding the last / at the end). Note that the redirect URI you set up in your backend must be an exact match of the one you have set up in your Credentials Page. Also please note that it might take some minutes to update this URI.
Moreover, this is a guide on how to create a Sign In button that will authorise your users that I believe will be useful for you.
I have a program that asks the user to type in a URL, and click download. Then the program downloads the webpage.
However, some websites use SSL, and in that case the user has to prefix his URL with https:// for this to work.
The problem is that the user may not know whether the website uses SSL, and may type http://... instead of https://....
Is there some way to send a preliminary message to the website (from vb.net) asking whether the URL should start with https or just http? If there is, I can correct the user URL before attempting to retrieve the web page.
(I should say there it is not enough to use something like this:
request.RequestUri.Scheme - this looks at the URL the user submitted, not the URL coming back from the server, as far as I know)
For websites that uses SSL, usually they will force the request to use HTTPS. That is when you send a request in HTTP, for example, http://www.example.com, the website will send a redirect response with HTTP status code 302 as well as the URL the client side that initiate the request should redirect user to.
So, you can try HTTP first and check the response to see if there is a redirect. So, you will need to handle that in your code.
From Google Developer Console page, it explains the meaning of "Google Redirect URI" is :
Determines where the response is sent. The value of this parameter
must exactly match one of the values registered in the APIs Console
(including the http or https schemes, case, and trailing '/').
Not sure, whether this explanation is same as my think or not:
On my app (web application), it allow user to sign-in via Google (Assume, I registered a Google project with ClientID/secrete configured on my web-app, and on Google COnsole page the redirectURI is "http://bbc.co.uk" ).
After I login to Google successfully by my Google account, my Web-app login page will be redirected to "http://bbc.co.uk" as on Google RedirectURI ?
That last sentence is a bit ambiguous. A better description is:
After I login to my Google Account successfully with my Google
credentials, Google will redirect my browser back to
Redirect URI http://bbc.co.uk that is registered for the web app in the Google > API Console.
Just to be sure: this means that the Redirect URI should be part of your web app. It is not actually Google's Redirect URI but it is the Redirect URI that you've configured in the Google API Console for your web app.
Let me see if i cant explain this in a different way.
What would happen if you didn't have a redirect URI?
If you go to the Developer console and create a Client ID for native application by default the redirect URI
This will redirect the Authentication back to where ever it was request from. In the case of native installed applications this is needed because there is no way to know where the user is that has installed the application.
Lets look at this in action
Request authentication:
https://accounts.google.com/o/oauth2/auth?client_id={clientid}.apps.googleusercontent.com&redirect_uri=urn:ietf:wg:oauth:2.0:oob&scope=https://www.googleapis.com/auth/analytics.readonly&response_type=code
If you place the above line in a web browser the user will see something like this
Assuming they select ok then they get this displayed back at them.
That is the Authentication code, not to be confused with the access token. This is just the first step. Once your application has the Authentication code it can then exchanged that for a refresh token and a access token. Most client library handle this step internally. You and your users never see this Authentication code.
Back Redirect URI:
In the case if a web application your web application needs to be able to get a hold of that authentication code. Now I probably shouldn't be telling you this but you could just use a native application client in your web application and it will work. Why because the authentication server will just return the Authentication code back to where ever it was that requested it. For security reasons this is probably a bad practice.
So what do you need to place in the redirect uri. It is the file that you have ready to handle this authentication code. Normally its just the same as the login page, but you can have several. Just put each of them on separate lines.
http://localhost/login.php
http://www.daimto.com/login.php
I like to do this because then it works when i am testing locally vs when its up on the server.
I think the wording "Determines where the response is sent" confuses a lot of beginners as it implies that Google will somehow call that URL directly. Then the word "redirect" leads people to confuse the URL with the page of the app that the user will get to once he's authenticated.
http://bbc.co.uk is definitely NOT the redirect URL. A redirect URL would normally look like https://myapp.com/servlet_or_php_that_i_wrote_to_process_tokens.php
Google will "call" servlet_or_php_that_i_wrote_to_process_tokens.php by redirecting the browser to it. Once servlet_or_php_that_i_wrote_to_process_tokens.php as done whatever is needed with the token, it will then issue its own redirect to (say) bbc.co.uk
I am trying to optimize my site for all HTTPS. I know that Twitter is all HTTPS and I noticed that they don't redirect HTTP to HTTPS, but instead just initiate an HTTPS connection.
Here is a screenshot of Google Chrome's Network Activity, notice there is no redirect (301/302), the HTTP request (first line) just hangs as pending and the second line is the HTTPS page. Note, I have cleared all my browser cache so HTTP Strict Transport Security (HSTS) shouldn't matter.
Here is another screenshot of the request/response for the HTTPS page. Notice, that it seems Twitter inserts some fields into the REQUEST, such as :scheme
How do they do this? I would assume its faster so that if a user types twitter.com into their browser, instead of a redirect (think extra network round trip), Twitter seems to seamlessly move to SSL (HTTPS).
A follow on question would be, does this work in all browsers?
They have been added to a list of preloaded HSTS sites in Chrome and Mozilla Firefox.
I am making app that takes a screenshot of a URL requested by the user. I want to make it as transparent as possible when sites that require username and passwords are in question.
For instance, if user wants to screenshot its iGoogle page, he will send the server URL but, the screenshot will not be the same as what he sees on his screen.
Is there any way to do this ? I guess that in such cases I will have to actually request screenshot from the user. Perhaps user can even deliver me his cookie for that domain.
Any thoughts ?
Ty.
Yes, in most cases you'll need user's cookies.
If site uses regular cookies, you can create bookmarklet that reads document.cookie. This will not work with httpOnly cookies which are used increasingly often for sessions.
Some sites limit sessions to certain IP, and in that case you can't take screenshot without proxying request through user's computer.
If you can get user to use bookmarlet, an interesting trick would be to read and send DOM to your server:
image.src = 'http://example.com?source=' +
escape(document.documentElement.innerHTML);
For HTTP authentication easiest solution would be to ask user for login/password.