Authenticate to an API with OWASP ZAP without using OpenAPI or Swagger specs - authentication

I am trying to authenticate to my API to perform some passive/active scan using OWASP ZAP.
I don't have any Swagger or OpenAPI specification, but I have some HTTP tests (Javascript) that might help. However, I can not figure out to authenticate to my API with ZAP.
Ideally I would like to automate the scan given some specs (in whatever format ZAP understands, but not automatic tools like OpenAPI Swagger), a URL entry-point and username/password, but I am stuck with more fundamental steps like the authentication.
I've been following this guide: https://www.zaproxy.org/docs/desktop/ui/dialogs/session/context-auth/
I added a username/password pair in Session > Contexts > Default Context > Users:
I then provided the details about the authentication API endpoint in Sessio > Context > Default Context > Authentication:
I made sure the button is pushed for "Forced User Mode enabled" (see the red circle in the previous screenshot on the "user details".
Then I right clicked on my "Default Context" (I created this with this name, nothing to do with ZAP terminology, it's just a ZAP Context) and clicked on "Active Scan".
Then I clicked on the button "Start Scan" from the pop-up window after the right click.
Then nothing happens. I don't see anything moving or logging or blinking.
How do I authenticate to my API with OWASP ZAP?
How do I reuse the JWT token to be used in other HTTP requests as an header?
Is there a way I could mime what I do in the HTTP integration tests to let ZAP discover issues with HTTP paths, HTTP query parameters and so on?
How can I export the above into a script I could invoke from the command line?
EDIT 1
This is not a website with HTML or a webapp. This is just a REST API via HTTPS with requests/responses, paths, query parameters and headers.
The "Include Context" does not contain any URL. However I just tried again adding the URL I specified in the "Authentication" menu, then tried again with the "Active Scan" and nothing happens.
The login/logout regex in "Authentication" were not containing anything because there is no such "logout" - the JWT token just expires and the request to any API endpoint is not valid any more.
Anyway I added the HTTP path (not the protocol or the domain/host, just the path without any /) for the "login" URL to both the regex fields in the form on ZAP. Then tried again "Active Scan" and nothing happens.
What is this "Active Scan" supposed to do? Do I have to provide all the valid paths? What about the query parameters?
How can I obtain the JWT token from the "login" API endpoint (it's not a webpage), and reuse it as an HTTP header during the scan?
How do I passively scan any endpoint starting from e.g. the "login" API endpoint (again - not a webpage, just a REST endpoint).
EDIT 2
I am using ZAP 2.9.0 via snap on Linux, there is no other version available:
$ snap find zaproxy
Name Version Publisher Notes Summary
zaproxy 2.9.0 psiinon classic OWASP ZAP, a tool for finding vulnerabilities in web applications
$ snap install zaproxy --classic
zaproxy 2.9.0 from Simon Bennetts (psiinon) installed
EDIT 3
I added this regex https?:\/\/example.org\/.* (with my host, not "example") to:
the "Include in Context" menu item
the "Authentication" menu item in both "login" and "logout" regex patterns.
Then tried again the "Active Scan": nothing happens - no output in the tabs on the lower part of the window, no logs, no blinking semaphores.

So you seem to have missed two key details.
You didn't mention what's included in the Context, or that you'd configured Include in Context pattern(s).
Also looking at your screenshot you haven't identified logged in or logged out identifiers so zap can never know what state it is in.
How can I obtain the JWT token from the "login" API endpoint (it's not a webpage), and reuse it as an HTTP header during the scan?
In Zap 2.10 you can also setup auth polling as a verification strategy.
If you have a non-standard auth mechanism then there are various options, such as using the Replacer add-on or an HttpSender script to set/update a header/token value. As of 2.10.0 this can also be done via env vars:
https://www.zaproxy.org/docs/desktop/start/features/authentication/
What is this "Active Scan" supposed to do? Do I have to provide all the valid paths? What about the query parameters?
This is why importing OpenAPI etc is valuable. Other options are to proxy functional tests. Zap has to be aware of the content/functionality in order to test it effectively.
How do I passively scan any endpoint starting from e.g. the "login" API endpoint (again - not a webpage, just a REST endpoint).
Passive scanning takes place on proxied or spidered traffic.
The "Include Context" does not contain any URL. However I just tried again adding the URL I specified in the "Authentication" menu, then tried again with the "Active Scan" and nothing happens.
Include in Context should be a regex pattern that will match your endpoints, unless you are literally only testing a single specific URL. Ex: https?:\/\/example.org/.* (.* being regex wildcard and matching anything on example.org/)

Related

Trying to get Client ID and Client Secret for Yahoo API but no option to select "Installed Application"

I've been attempting to utilize the Yahoo API to connect to fantasy sports data. I don't have a web URL to connect to when I attempt to create an app, and everything I've read has mentioned that I should select "Installed Application" to avoid using a URL.
This option is not available to me.
Yahoo Developer Screenshot
On the official documentation this is what I see here
Yahoo Developer Documentation
Not sure if it makes a difference but I'm based out of Canada.
Thanks!
It's hard to find the documentation for it, but what you want is called oob (out of bounds, out of band). To get this to work, the steps are a little different (there is no such thing as installed application from what I'm aware of):
Add oob as one of the redirect uris
Make your normal request to the auth/login uri
This will give you a new page that provides you with a token. At this point, you need to manually use this token when you request the /auth/token uri
This last step just replaces the redirect to https://your-redirect.com/token and you will end up in the same position as before.
Now, you still shouldn't put your client_secret in your application if you plan on releasing it, but if you're using it internally or letting others provide their own secret then you should be good.
If you find the original documentation with the PHP source code, you can follow this flow.

Cannot add http url to Authorized redirect URIs in gcloud's OAuth 2.0 client IDs

I have a webapp that uses google authentication. When I click on the login, it gets the following error:
The redirect URI in the request, http://campus-study-2019.appspot.com/oauth2callback, does not match the ones authorized for the OAuth client. To update the authorized redirect URIs, visit: https://console.developers.google.com/apis/credentials/oauthclient/477760217336-q6bdhcdqvsrf56kaa7k0ifennf775lug.apps.googleusercontent.com?project=477760217336
When I try to add
http://campus-study-2019.appspot-preview.com/oauth2callback to my Authorized redirect URIs in the Google Cloud platforms. I get the following:
Invalid Redirect: You are using a sensitive scope. URI must use https:// as the scheme.
Does anyone have any idea?
However, previously (1-2 months ago) I could easily add http in the list of authorized redirect URIs
Make sure to check the protocol "http://" or "https://" as google checks protocol as well. Better to add both URL in the list.
& or &?
trailing slash(/) or open ? (CMD/CTRL)+F, search for the exact match in the credential page. If not found then search for the missing one.
Wait until google refresh it. May happen in each half an hour if you are changing frequently or it may stay in the pool.
For your reference, please check the below link where you could check other options about your issue:
Google OAuth 2 authorization - Error: redirect_uri_mismatch
The error is due to there being at least one restricted scope added to the project where the OAuth client ID resides.
You can see the set scopes for Google APIs by going to your project in the Developers Console -> APIs & Services -> OAuth consent screen. To fix the error there are two options:
Remove the restricted scopes from the project (there will be a warning icon next to them on the OAuth consent screen page)
Add only HTTPS URLs as authorized restricted URIs (https://campus-study-2019.appspot-preview.com/oauth2callback in this case)

Jmeter is not passing the Microsoft authentication in my script for testing performance on my test website, showing Access denied

My application has a microsoft authentication on it before logging into it & I have recorded script but when I am running it, it is showing me access denied error everytime.I have set authentication manager but still same error. See image attached.
I have tried HTTP Authentication Manager & provided login username & password.
Most probably your application uses OAuth therefore it is neither something you can really record and replay nor handle with the HTTP Authorization Manager.
Depending on your application setup you will either need:
To perform correlation of the query parameters
Or to pass the relevant Authorization Bearer token via HTTP Header Manager. The process of obtaining the token can be different depending on your application login chain implementation, check out How to Run Performance Tests on OAuth Secured Apps with JMeter article to get a couple of ideas regarding bypassing 3rd-party provider login challenge in JMeter tests.
Check if you can provide the auth credentials as parameter of the requests.
for example www.abc.com?username=abc&password=abc. Replicate the same with Jmeter
Use Fiddler (or you can get away with browser dev tools if you don't mind searching manually) and log in manually via your browser.
Check the request(s) that are submitted to Microsoft for tokens/GUIDs and search for where the browser got those strings from (it'll be in one of the previous requests' responses' bodies or redirect URLs. In Fiddler you can use the find function on responses, browser dev tools you'll have to find it manually).
You can then use a JMeter Regular Expression Extractor post-processor (or any of the other post processors you prefer) to extract that string from the earlier request into a variable.
In your login request you can then use the value of that variable (if you used regular expression post-processor with a capture group the first group's value will be ${variable_g1}
You'll probably have an anti-forgery value that you can extract from the HTML of the login page which needs to be submitted with the username and password and then in the response you'll get a cookie set and potentially JWT token in the response body/URL.

Anti forgery token and web testing

I am trying to do a web test in VS2012 for an MVC site.
One of the scenarios is to login and go through a list of products, select the one you want and follow through to the purchase page.
Problem is that when the web test is run, I get an error about the anti forgery token and that it does not match.
How on earth is it possible to do the testing with the anti forgery token? The user must login - there will be thousands of users for the load test (eventually) but need to make it work for 1 user first.
the login view/action does do an AntiForgeryToken in the view and validation on the controller.
any advice and tips are appreciated.
Once you run your script and it fails, go to the call proceeding the one that fails.
Go to the response tab
In the body, find the __RequestVerificationToken name which is in an input tag and extract everything in between the value attribute.
Select the value and right click > add extraction rule and press OK.
You will find an Extraction rules folder and underneath it, the Extraction rule we just created. Feel free to rename the Context Parameter Name.
Go to the next page , which should be the one that failed, and find the Form Post Parameter named "__RequestVerificationToken". View it's properties
Bind it to the Context Parameter Name created previously. To do so, view the properties of this post parameter and set the "Value" to be:
{{Name Of Context Parameter}}
(Include the 2x curly braces)
Press enter to confirm/save
Next time you run the script - all works
This is how it worked for me...
I was seeing a similar problem. After recording a web test script, the script would fail at the point of log-in on with the following message:
The provided anti-forgery token was meant for user "Domain\UserName", but the current user is "".
The solution was to set the PreAuthenticate property to false in the test properties. By default the web tests will pass an authentication header to the server which was being used in the generation the token.
I am not familiar with "web testing in VS2012" but as I know "Anti-Forgery Token" requires sending the token from browser to the server back.
I had an experience with Selenium-Webdriver and suggest you use it because it provides an API to interact with supported browsers as real user does.
You can easily start using Selenium WebDriver if you add Selenium WebDriver 2.37.0 NuGet package to you test project.
Selenium-WebDriver makes direct calls to the browser using each
browser’s native support for automation. How these direct calls are
made, and the features they support depends on the browser you are
using.
Selenium-webdriver currently supports the following drivers:
Chrome
Internet Explorer
Firefox
Opera
HtmlUnit
Android

Google OAuth 2 authorization - Error: redirect_uri_mismatch

On the website https://code.google.com/apis/console I have registered my application, set up generated Client ID: and Client Secret to my app and tried to log in with Google.
Unfortunately, I got the error message:
Error: redirect_uri_mismatch
The redirect URI in the request: http://127.0.0.1:3000/auth/google_oauth2/callback did not match a registered redirect URI
scope=https://www.googleapis.com/auth/userinfo.profile https://www.googleapis.com/auth/userinfo.email
response_type=code
redirect_uri=http://127.0.0.1:3000/auth/google_oauth2/callback
access_type=offline
approval_prompt=force
client_id=generated_id
What does mean this message, and how can I fix it?
I use the gem omniauth-google-oauth2.
The redirect URI (where the response is returned to) has to be registered in the APIs console, and the error is indicating that you haven't done that, or haven't done it correctly.
Go to the console for your project and look under API Access. You should see your client ID & client secret there, along with a list of redirect URIs. If the URI you want isn't listed, click edit settings and add the URI to the list.
EDIT: (From a highly rated comment below) Note that updating the google api console and that change being present can take some time. Generally only a few minutes but sometimes it seems longer.
In my case it was www and non-www URL. Actual site had www URL and the Authorized Redirect URIs in Google Developer Console had non-www URL. Hence, there was mismatch in redirect URI. I solved it by updating Authorized Redirect URIs in Google Developer Console to www URL.
Other common URI mismatch are:
Using http:// in Authorized Redirect URIs and https:// as actual URL, or vice-versa
Using trailing slash (http://example.com/) in Authorized Redirect URIs and not using trailing slash (http://example.com) as actual URL, or vice-versa
Here are the step-by-step screenshots of Google Developer Console so that it would be helpful for those who are getting it difficult to locate the developer console page to update redirect URIs.
Go to https://console.developers.google.com
Select your Project
Click on the menu icon
Click on API Manager menu
Click on Credentials menu. And under OAuth 2.0 Client IDs, you will find your client name. In my case, it is Web Client 1. Click on it and a popup will appear where you can edit Authorized Javascript Origin and Authorized redirect URIs.
Note: The Authorized URI includes all localhost links by default, and any live version needs to include the full path, not just the domain, e.g. https://example.com/path/to/oauth/url
Here is a Google article on creating project and client ID.
If you're using Google+ javascript button, then you have to use postmessage instead of the actual URI. It took me almost the whole day to figure this out since Google's docs do not clearly state it for some reason.
In any flow where you retrieved an authorization code on the client side, such as the GoogleAuth.grantOfflineAccess() API, and now you want to pass the code to your server, redeem it, and store the access and refresh tokens, then you have to use the literal string postmessage instead of the redirect_uri.
For example, building on the snippet in the Ruby doc:
client_secrets = Google::APIClient::ClientSecrets.load('client_secrets.json')
auth_client = client_secrets.to_authorization
auth_client.update!(
:scope => 'profile https://www.googleapis.com/auth/drive.metadata.readonly',
:redirect_uri => 'postmessage' # <---- HERE
)
# Inject user's auth_code here:
auth_client.code = "4/lRCuOXzLMIzqrG4XU9RmWw8k1n3jvUgsI790Hk1s3FI"
tokens = auth_client.fetch_access_token!
# { "access_token"=>..., "expires_in"=>3587, "id_token"=>..., "refresh_token"=>..., "token_type"=>"Bearer"}
The only Google documentation to even mention postmessage is this old Google+ sign-in doc. Here's a screenshot and archive link since G+ is closing and this link will likely go away:
It is absolutely unforgivable that the doc page for Offline Access doesn't mention this. #FacePalm
For my web application i corrected my mistake by writing
instead of : http://localhost:11472/authorize/
type : http://localhost/authorize/
Make sure to check the protocol "http://" or "https://" as google checks protocol as well.
Better to add both URL in the list.
1.you would see an error like this
2.then you should click on request details
after this , you have to copy that url and add this on https://console.cloud.google.com/
go to https://console.cloud.google.com/
click on Menu -> API & Services -> Credentials
you would see a dashboard like this ,click on edit OAuth Client
now in Authorized Javascript Origins and Authorized redirect URLS
add the url that has shown error called redirect_uri_mismatch i.e here it is
http://algorithammer.herokuapp.com , so i have added that in both the places in
Authorized Javascript Origins and Authorized redirect URLS
click on save and wait for 5 min and then try to login again
This seems quite strange and annoying that no "one" solution is there.
for me http://localhost:8000 did not worked out but http://localhost:8000/ worked out.
This answer is same as this Mike's answer, and Jeff's answer, both sets redirect_uri to postmessage on client side. I want to add more about the server side, and also the special circumstance applying to this configuration.
Tech Stack
Backend
Python 3.6
Django 1.11
Django REST Framework 3.9: server as API, not rendering template, not doing much elsewhere.
Django REST Framework JWT 1.11
Django REST Social Auth < 2.1
Frontend
React: 16.8.3, create-react-app version 2.1.5
react-google-login: 5.0.2
The "Code" Flow (Specifically for Google OAuth2)
Summary: React --> request social auth "code" --> request jwt token to acquire "login" status in terms of your own backend server/database.
Frontend (React) uses a "Google sign in button" with responseType="code" to get an authorization code. (it's not token, not access token!)
The google sign in button is from react-google-login mentioned above.
Click on the button will bring up a popup window for user to select account. After user select one and the window closes, you'll get the code from the button's callback function.
Frontend send this to backend server's JWT endpoint.
POST request, with { "provider": "google-oauth2", "code": "your retrieved code here", "redirect_uri": "postmessage" }
For my Django server I use Django REST Framework JWT + Django REST Social Auth. Django receives the code from frontend, verify it with Google's service (done for you). Once verified, it'll send the JWT (the token) back to frontend. Frontend can now harvest the token and store it somewhere.
All of REST_SOCIAL_OAUTH_ABSOLUTE_REDIRECT_URI, REST_SOCIAL_DOMAIN_FROM_ORIGIN and REST_SOCIAL_OAUTH_REDIRECT_URI in Django's settings.py are unnecessary. (They are constants used by Django REST Social Auth) In short, you don't have to setup anything related to redirect url in Django. The "redirect_uri": "postmessage" in React frontend suffice. This makes sense because the social auth work you have to do on your side is all Ajax-style POST request in frontend, not submitting any form whatsoever, so actually no redirection occur by default. That's why the redirect url becomes useless if you're using the code + JWT flow, and the server-side redirect url setting is not taking any effect.
The Django REST Social Auth handles account creation. This means it'll check the google account email/last first name, and see if it match any account in database. If not, it'll create one for you, using the exact email & first last name. But, the username will be something like youremailprefix717e248c5b924d60 if your email is youremailprefix#example.com. It appends some random string to make a unique username. This is the default behavior, I believe you can customize it and feel free to dig into their documentation.
The frontend stores that token and when it has to perform CRUD to the backend server, especially create/delete/update, if you attach the token in your Authorization header and send request to backend, Django backend will now recognize that as a login, i.e. authenticated user. Of course, if your token expire, you have to refresh it by making another request.
Oh my goodness, I've spent more than 6 hours and finally got this right! I believe this is the 1st time I saw this postmessage thing. Anyone working on a Django + DRF + JWT + Social Auth + React combination will definitely crash into this. I can't believe none of the article out there mentions this except answers here. But I really hope this post can save you tons of time if you're using the Django + React stack.
In my case, my credential Application type is "Other". So I can't find Authorized redirect URIs in the credentials page. It seems appears in Application type:"Web application". But you can click the Download JSON button to get the client_secret.json file.
Open the json file, and you can find the parameter like this: "redirect_uris":["urn:ietf:wg:oauth:2.0:oob","http://localhost"]. I choose to use http://localhost and it works fine for me.
When you register your app at https://code.google.com/apis/console and
make a Client ID, you get a chance to specify one or more redirect
URIs. The value of the redirect_uri parameter on your auth URI has to
match one of them exactly.
Checklist:
http or https?
& or &?
trailing slash(/) or open ?
(CMD/CTRL)+F, search for the exact match in the credential page. If
not found then search for the missing one.
Wait until google refreshes it. May happen in each half an hour if you
are changing frequently or it may stay in the pool. For my case it was almost half an hour to take effect.
for me it was because in the 'Authorized redirect URIs' list I've incorrectly put https://developers.google.com/oauthplayground/ instead of https://developers.google.com/oauthplayground (without / at the end).
The redirect url is case sensitive.
In my case I added both:
http://localhost:5023/AuthCallback/IndexAsync
http://localhost:5023/authcallback/indexasync
If you use this tutorial: https://developers.google.com/identity/sign-in/web/server-side-flow then you should use "postmessage".
In GO this fixed the problem:
confg = &oauth2.Config{
RedirectURL: "postmessage",
ClientID: ...,
ClientSecret: ...,
Scopes: ...,
Endpoint: google.Endpoint,
}
beware of the extra / at the end of the url
http://localhost:8000 is different from http://localhost:8000/
It has been answered thoroughly but recently (like, a month ago) Google stopped accepting my URI and it would not worked. I know for a fact it did before because there is a user registered with it.
Anyways, the problem was the regular 400: redirect_uri_mismatch but the only difference was that it was changing from https:// to http://, and Google will not allow you to register http:// redirect URI as they are production publishing status (as opposed to localhost).
The problem was in my callback (I use Passport for auth) and I only did
callbackURL: "/register/google/redirect"
Read docs and they used a full URL, so I changed it to
callbackURL: "https://" + process.env.MY_URL+ "/register/google/redirect"
Added https localhost to my accepted URI so I could test locally, and it started working again.
TL;DR use the full URL so you know where you're redirecting
2015 July 15 - the signin that was working last week with this script on login
<script src="https://apis.google.com/js/platform.js" async defer></script>
stopped working and started causing Error 400 with Error: redirect_uri_mismatch
and in the DETAILS section: redirect_uri=storagerelay://...
i solved it by changing to:
<script src="https://apis.google.com/js/client:platform.js?onload=startApp"></script>
Rails users (from the omniauth-google-oauth2 docs):
Fixing Protocol Mismatch for redirect_uri in Rails
Just set the full_host in OmniAuth based on the Rails.env.
# config/initializers/omniauth.rb
OmniAuth.config.full_host = Rails.env.production? ? 'https://domain.com' : 'http://localhost:3000'
REMEMBER: Do not include the trailing "/"
None of the above solutions worked for me. below did
change authorised Redirect urls to - https://localhost:44377/signin-google
Hope this helps someone.
My problem was that I had http://localhost:3000/ in the address bar and had http://127.0.0.1:3000/ in the console.developers.google.com
Just make sure that you are entering URL and not just a domain.
So instead of:
domain.com
it should be
domain.com/somePathWhereYouHadleYourRedirect
Anyone struggling to find where to set redirect urls in the new console: APIs & Auth -> Credentials -> OAuth 2.0 client IDs -> Click the link to find all your redirect urls
My two cents:
If using the Google_Client library do not forget to update the JSON file on your server after updating the redirect URI's.
I also get This error Error-400: redirect_uri_mismatch
This is not a server or Client side error but you have to only change by checking that you haven't to added / (forward slash) at the end like this
redirecting URL list ❌:
https://developers.google.com/oauthplayground/
Do this only ✅:
https://developers.google.com/oauthplayground
Let me complete #Bazyl's answer: in the message I received, they mentioned the URI
"http://localhost:8080/"
(which of course, seems an internal google configuration). I changed the authorized URI for that one,
"http://localhost:8080/" , and the message didn't appear anymore... And the video got uploaded... The APIS documentation is VERY lame... Every time I have something working with google apis, I simply feel "lucky", but there's a lack of good documentation about it.... :( Yes, I got it working, but I don't yet understand neither why it failed, nor why it worked... There was only ONE place to confirm the URI in the web, and it got copied in the client_secrets.json... I don't get if there's a THIRD place where one should write the same URI... I find nor only the documentation but also the GUI design of Google's api quite lame...
I needed to create a new client ID under APIs & Services -> Credentials -> Create credentials -> OAuth -> Other
Then I downloaded and used the client_secret.json with my command line program that is uploading to my youtube account. I was trying to use a Web App OAuth client ID which was giving me the redirect URI error in browser.
I have frontend app and backend api.
From my backend server I was testing by hitting google api and was facing this error. During my whole time I was wondering of why should I need to give redirect_uri as this is just the backend, for frontend it makes sense.
What I was doing was giving different redirect_uri (though valid) from server (assuming this is just placeholder, it just has only to be registered to google) but my frontend url that created token code was different. So when I was passing this code in my server side testing(for which redirect-uri was different), I was facing this error.
So don't do this mistake. Make sure your frontend redirect_uri is same as your server's as google use it to validate the authenticity.
The main reason for this issue will only come from chrome and chrome handles WWW and non www differently depending on how you entered your URL in the browsers and it searches from google and directly shows the results, so the redirection URL sent is different in a different case
Add all the possible combinations you can find the exact url sent from fiddler , the 400 error pop up will not give you the exact http and www infromation
Try to do these checks:
Bundle ID in console and in your application. I prefer set Bundle ID of application like this "org.peredovik.${PRODUCT_NAME:rfc1034identifier}"
Check if you added URL types at tab Info just type your Bundle ID in Identifier and URL Schemes, role set to Editor
In console at cloud.google.com "APIs & auth" -> "Consent screen" fill form about your application. "Product name" is required field.
Enjoy :)