I am writing an email DKIM signature verifier program in java. My problem is a special case:
1 – I don't have any issue in creating body hash. My problem is in signing canonicalized header.
2 – I use a gmail account to created test emails and I verify their header signature properly in many cases. But only some special scenarios fail. I do many tests and recognized when 'h' tag in ' DKIM-Signature' header has 2 lines, signature verification fails.
This is an example of failing header:
...
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20161025;
h=mime-version:references:in-reply-to:from:date:message-id:subject:to
:cc;
...
For clarification I guess my problem is here :
h=mime-version:references:in-reply-to:from:date:message-id:subject:to
:cc;
Every time 'h' tag unfolded to 2 lines verification fails. I am familiar with line unfolding rules and I can correctly verify signature of headers that unfolded in other positions.
After header canonicalization, the content that should verify signature is:
...
dkim-signature:v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to:cc; bh=pr1Zv1pbjuW/HrdQ6zZ5mYq51Bi+uidXLGOQ265rvEs=; b=
I test many scenarios to modify signing content, hope to correctly verification, include inserting a 'space' in line folding position and signing content become such as :
... h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; ...
I know my question is very specific. But hope someone have experience in DKIM signing and its special cases.
Note that the DKIM header you posted is incomplete (bh and b tag missing).
But the second (relaxed) canonicalization of the DKIM header there you added a space in ...:to :cc;... should be correct. If it still does not verify, take a closer look at the rest of the signed headers too. Maybe there is another error.
The unfolding itself does not remove any whitespace besides the CRLF. From https://www.rfc-editor.org/rfc/rfc5322#section-2.2.3
Unfolding is accomplished by simply removing any CRLF
that is immediately followed by WSP.
So after the unfloding, you are left with ...:to :cc;...
In the next step, this is reduced to a single space,
as per https://www.rfc-editor.org/rfc/rfc6376#section-3.4.2
Convert all sequences of one or more WSP characters to a single SP character.
Related
I'm trying to authorize a user depending on his group membership.
Apache is configured as follows:
AuthLDAPURL "ldap://DOMAIN.COM/DC=FOO,DC=COM?CN?SUB?(objectClass=user)" NONE
AuthLDAPBindDN "CN=..."
AuthLDAPBindPassword "..."
Require ldap-group CN=\#\#SOMETEXT,OU=GSI,OU=DMM,OU=DDSIS,OU=Admin_Exchange,DC=DOMAIN,DC=com
The problem comes from the two hashes that appear in the CN.
I tried with and without quotes, with two backslashes and no backslashes, URL encode, but nothing worked.
I always get the following error message.
[Thu Feb 13 18:40:56.728349 2020] [authnz_ldap:debug] [pid 17154]
mod_authnz_ldap.c(922): [client 10.255.180.148:65050] AH01719:
auth_ldap authorize: require group
"CN=##SOMETEXT,OU=GSI,OU=DMM,OU=DDSIS,OU=Admin_Exchange,DC=DOMAIN,DC=com": didn't match with attr Comparison complete [member][34 - Invalid DN
syntax]
When i remove the hashes, i get a no such object error, that confirms the problem comes from the hashes.
EDIT:
As suggested in the answer, I tried to write the octal representation of the hash character \43 but it gave me the exact same error:
"CN=##SOMETEXT, ...Invalid DN syntax]
So that doesn't seem to change what Apache sends to LDAP
You typically can get around these type of issues with something like:
CN=\23\23SOMETEXT,OU=GSI,OU=DMM,OU=DDSIS,OU=Admin_Exchange,DC=DOMAIN,DC=com
As described in RFC 4514 and Characters to Escape
However, different applications may be trying to parsing these parameters for you and may interfere with the escaping process.
The RIGHT thing to do is rename the group.
The problem you are encountering will persist with other application in the future. Best Practices for LDAP imply all Relative Distinguished Names be "URL Safe" and not require escaping.
You will need to escape the string according to RFC 4515 String Representation of Search Filters
Generally, you need to escape the items listed in RFC 4515 String Representation of Search Filters and I would suggest, also any non-UTF8 character.
I also found some methods that may be helpful to get you started.
I believe the proper escaped value you are trying to find is: All in 463"567y\5c22"\c2\a4&/2#%&! Test Group
Finally, quit it. Start populating a searching for Description or some other non-naming attribute. (any attribute that is not part of the DN) Make your DNs never changing. No user should ever see a DN which should be only a path to an entry. You will have issues with many "off-the-shelve" tools if you continue this practice.
I tried and was not even able to create an entry in two different vendors' tools.
I am coding some test software to simulate something like a router. It will send URL requests on behalf of multiple users.
Is there any HTTP GET header field which I can send which the receiving server will always send back to me unchanged in the response so that I can associate the response with a user?
This is test software for use on a local LAN only, so I don't mind misusing a field, just as long as I get it returned unchanged.
according to http 1.1 rfc, response is:
Response = Status-Line ; Section 6.1
*(( general-header ; Section 4.5
| response-header ; Section 6.2
| entity-header ) CRLF) ; Section 7.1
CRLF
[ message-body ] ; Section 7.2
and here is notation:
*rule
The character "*" preceding an element indicates repetition. The
full form is "<n>*<m>element" indicating at least <n> and at most
<m> occurrences of element. Default values are 0 and infinity so
that "*(element)" allows any number, including zero; "1*element"
requires at least one; and "1*2element" allows one or two.
[rule]
Square brackets enclose optional elements; "[foo bar]" is
equivalent to "*1(foo bar)".
so, the only requirement for server is to respond with status code, other components are optional, always, which effectively means there is no requirement to send any header back
also, this contains list of all possible headers, none of them meet your requirements
I'm not sure about http 2.0, maybe somebody could add information about it
I've read conflicting and somewhat ambiguous replies to the question "How is a multipart HTTP request content length calculated?". Specifically I wonder:
What is the precise content range for which the "Content-length" header is calculated?
Are CRLF ("\r\n") octet sequences counted as one or two octets?
Can someone provide a clear example to answer these questions?
How you calculate Content-Length doesn't depend on the status code or media type of the payload; it's the number of bytes on the wire. So, compose your multipart response, count the bytes (and CRLF counts as two), and use that for Content-Length.
See: http://httpwg.org/specs/rfc7230.html#message.body.length
The following live example should hopefully answer the questions.
Perform multipart request with Google's OAuth 2.0 Playground
Google's OAuth 2.0 Playground web page is an excellent way to perform a multipart HTTP request against the Google Drive cloud. You don't have to understand anything about Google Drive to do this -- I'll do all the work for you. We're only interested in the HTTP request and response. Using the Playground, however, will allow you to experiment with multipart and answer other questions, should the need arise.
Create a test file for uploading
I created a local text file called "test-multipart.txt", saved somewhere on my file system. The file is 34 bytes large and looks like this:
We're testing multipart uploading!
Open Google's OAuth 2.0 Playground
We first open Google's OAuth 2.0 Playground in a browser, using the URL https://developers.google.com/oauthplayground/:
Fill in Step 1
Select the Drive API v2 and the "https://www.googleapis.com/auth/drive", and press "Authorize APIs":
Fill in Step 2
Click the "Exchange authorization code for tokens":
Fill in Step 3
Here we give all relevant multipart request information:
Set the HTTP Method to "POST"
There's no need to add any headers, Google's Playground will add everything needed (e.g., headers, boundary sequence, content length)
Request URI: "https://www.googleapis.com/upload/drive/v2/files?uploadType=multipart"
Enter the request body: this is some meta-data JSON required by Google Drive to perform the multipart upload. I used the following:
{"title": "test-multipart.txt", "parents": [{"id":"0B09i2ZH5SsTHTjNtSS9QYUZqdTA"}], "properties": [{"kind": "drive#property", "key": "cloudwrapper", "value": "true"}]}
At the bottom of the "Request Body" screen, choose the test-multipart.txt file for uploading.
Press the "Send the request" button
The request and response
Google's OAuth 2.0 Playground miraculously inserts all required headers, computes the content length, generates a boundary sequence, inserts the boundary string wherever required, and shows us the server's response:
Analysis
The multipart HTTP request succeeded with a 200 status code, so the request and response are good ones we can depend upon. Google's Playground inserted everything we needed to perform the multipart HTTP upload. You can see the "Content-length" is set to 352. Let's look at each line after the blank line following the headers:
--===============0688100289==\r\n
Content-type: application/json\r\n
\r\n
{"title": "test-multipart.txt", "parents": [{"id":"0B09i2ZH5SsTHTjNtSS9QYUZqdTA"}], "properties": [{"kind": "drive#property", "key": "cloudwrapper", "value": "true"}]}\r\n
--===============0688100289==\r\n
Content-type: text/plain\r\n
\r\n
We're testing multipart uploading!\r\n
--===============0688100289==--
There are nine (9) lines, and I have manually added "\r\n" at the end of each of the first eight (8) lines (for readability reasons). Here are the number of octets (characters) in each line:
29 + '\r\n'
30 + '\r\n'
'\r\n'
167 + '\r\n'
29 + '\r\n'
24 + '\r\n'
'\r\n'
34 + '\r\n' (although '\r\n' is not part of the text file, Google inserts it)
31
The sum of the octets is 344, and considering each '\r\n' as a single one-octet sequence gives us the coveted content length of 344 + 8 = 352.
Summary
To summarize the findings:
The multipart request's "Content-length" is computed from the first byte of the boundary sequence following the header section's blank line, and continues until, and includes, the last hyphen of the final boundary sequence.
The '\r\n' sequences should be counted as one (1) octet, not two, regardless of the operating system you're running on.
If an http message has Content-Length header, then this header indicates exact number of bytes that follow after the HTTP headers. If anything decided to freely count \r\n as one byte then everything would fall apart: keep-alive http connections would stop working, as HTTP stack wouldn't be able to see where the next HTTP message starts and would try to parse random data as if it was an HTTP message.
\n\r are two bytes.
Moshe Rubin's answer is wrong. That implementation is bugged there.
I sent a curl request to upload a file, and used WireShark to specifically harvest the exact actual data sent by my network. A methodology that everybody should agree is more valid than on online application somewhere gave me a number.
--------------------------de798c65c334bc76\r\n
Content-Disposition: form-data; name="file"; filename="requireoptions.txt"\r\n
Content-Type: text/plain\r\n
\r\n
Pillow
pyusb
wxPython
ezdxf
opencv-python-headless
\r\n--------------------------de798c65c334bc76--\r\n
Curl, which everybody will agree likely implemented this correctly:
Content-Length: 250
> len("2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d646537393863363563333334626337360d0a436f6e74656e742d446973706f736974696f6e3a20666f726d2d646174613b206e616d653d2266696c65223b2066696c656e616d653d22726571756972656f7074696f6e732e747874220d0a436f6e74656e742d547970653a20746578742f706c61696e0d0a0d0a50696c6c6f770d0a70797573620d0a7778507974686f6e0d0a657a6478660d0a6f70656e63762d707974686f6e2d686561646c6573730d0a2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d2d646537393863363563333334626337362d2d0d0a")
500
(2x250 = 500, copied the hex stream out of WireShark.)
I took the actual binary there. The '2d' is --- which starts the boundary.
Please note, giving the wrong count to the server treating 0d0a as 1 rather than 2 octets (which is insane they are octets and cannot be compound), actively rejected the request as bad.
Also, this answers the second part of the question. The actual Content Length is everything here. From the first boundary to the last with the epilogue --\r\n, it's all the octets left in the wire.
EDIT: The NHTSA docs, as CBroe points out, say to replace an ampersand with an underscore. However, I'm also getting an error with forward slashes (albeit a different one, page not found, because it's decoding the slash), for example the make 'EXISS/SOONER':
http://www.nhtsa.gov/webapi/api/Recalls/vehicle/modelyear/1997/make/EXISS%2FSOONER?format=json
And replacing the ampersand with an underscore no longer results in an error message, but in zero results returned, which should not be the case.
ORIGINAL POST:
I'm trying to download the content from the following URL:
http://www.nhtsa.gov/webapi/api/Recalls/vehicle/modelyear/1997/make/s&s?format=json
And the site returns the following error message:
Server Error in '/' Application.
A potentially dangerous Request.Path value was detected from the client (&).
The problem is the ampersand; a similar request for a different car make works:
http://www.nhtsa.gov/webapi/api/Recalls/vehicle/modelyear/1997/make/toyota?format=json
I have verified from a different endpoint that S&S is a valid make for the API.
Based on stackoverflow answers, I've tried all the following without success:
http://www.nhtsa.gov/webapi/api/Recalls/vehicle/modelyear/1997/make/s%26s?format=json
http://www.nhtsa.gov/webapi/api/Recalls/vehicle/modelyear/1997/make/s%26amp;s?format=json
http://www.nhtsa.gov/webapi/api/Recalls/vehicle/modelyear/1997/make/s%26amp%3Bs?format=json
I faced with one issue, which I can't understand in Freeradius users file.
My goal is just authenticate external user "shad" with password "test".
I added line in /etc/raddb/users the following line:
shad Cleartext-Password == "test"
Result was Reject. If I change "==" operator to ":=" Authentication is successful.
So my question is the following:
Why I can't use "==" operator while FreeRadius documentation tells:
"Attribute == Value
As a check item, it matches if the named attribute is present in the request, AND has the given value."
And one more question.
In some resourses I faced with such lines:
shad Auth-Type := Local, User-Password == "test"
I tried and it doesn't work. Responce is Reject with log:
[pap] WARNING! No "known good" password found for the user. Authentication may fail because of this.
How the users file works
For the answer below, pairs is referring to Attribute Value Pairs (AVPs), that is, a tuple consisting of an attribute an operator and a value.
There are three lists of attribute(s) (pairs) that are accessible from the users file. These are bound to the request the server is currently processing, and they don't persist across multiple request/response rounds.
request - Contains all the pairs from the original request received from the NAS (Network Access Server) via the network.
control - Initially contains no pairs, but is populated with pairs that control how modules process the current request. This is done from the users file or unlang (the FreeRADIUS policy language used in virtual servers).
reply - Contains pairs you want to send back to the NAS via the network.
The users file module determines the list it's going to use for inserting/searching, by where the pair is listed in the entry, and the operator.
The first line of the entry contains check pairs that must match in order for the entry to be used. The first line also contains control pairs, those you want to be inserted into the control list if all the check pairs match.
Note: It doesn't matter which order the pairs are listed in. control pairs will not be inserted unless all the check pairs evaluate to true.
check and control pairs are distinguished by the operator used. If an assignment operator is used i.e. ':=' or '=' then the pair will be treated as a control pair. If an equality operator such as '>', '<', '==', '>=', '<=', '=~' is used, the pair will be treated as a check pair.
Subsequent lines in the same entry contain only reply pairs. If all check pairs match, reply pairs will be inserted into the reply list.
Cleartext-Password
Cleartext-Password is strictly a control pair. It should not be present in any of the other lists.
Cleartext-Password is one of a set of attributes, which should contain the 'reference' (or 'known good') password, that is, the local copy of the users password. An example of another pair in this set is SSHA-Password - this contains a salted SHA hash of the users password.
The reference password pairs are searched for (in the control list) by modules capable of with authenticating users using the 'User-Password' pair. In this case that module is 'rlm_pap'.
User-Password
User-Password is strictly a request pair. It should not be present in any of the other lists.
User-Password is included in the request from the NAS. It contains the plaintext version of the password the user provided to the NAS. In order to authenticate a user, a module needs to compare the contents of User-Password with a control pair like Cleartext-Password.
In a users file entry when setting reference passwords you'll see entries like:
my_username Cleartext-Password := "known_good_password"
That is, if the username matches the value on the left (my_username), then insert the control pair Cleartext-Password with the value "known_good_password".
To answer the first question the reason why:
shad Cleartext-Password == "test"
Does not work, it is because you are telling the files module to search in the request list, for a pair which does not exist in the request list, and should never exist in the request list.
You might now be thinking oh, i'll use User-Password == "test" instead, and it'll work. Unfortunately it won't. If the password matches then the entry will match, but the user will still be rejected, see below for why.
Auth-Type
Auth-Type is strictly a control pair. It should not be present in any of the other lists.
There are three main sections in the server for dealing with requests 'authorize', 'authenticate', 'post-auth'.
authorize is the information gathering section. This is where database lookups are done to authorise the user, and to retrieve reference passwords. It's also where Auth-Type is determined, that is, the type of authentication we want to perform for the user.
Authenticate is where a specific module is called to perform authentication. The module is determined by Auth-Type.
Post-Auth is mainly for logging, and applying further policies, the modules run in Post-Auth are determined by the response returned from the module run in Authenticate.
The modules in authorize examine the request, and if they think they can authenticate the user, and Auth-Type is not set, they set it to themselves. For example the rlm_pap module will set Auth-Type = 'pap' if it finds the User-Password in the request.
If no Auth-Type is set the request will be rejected.
So to answer your second question, you're forcing pap authentication, which is wrong, you should let rlm_pap set the Auth-Type by listing pap after the files module in the authorize section.
When rlm_pap runs in authenticate, it looks for a member of the set of 'reference' passwords described above, and if it can't find one, it rejects the request, this is what's happening above.
There's also a 'magic' Auth-Type, 'Accept', which skips the authenticate section completely and just accepts the user. If you want the used to do cleartext password comparison without rlm_pap, you can use:
shad Auth-Type := Accept, User-Password == "test"