Hash computation in Google safe browsing V2 implementation - api

I am trying to test my implementation of the google safe browsing api version 2.
To test a part of my code that sends requests for full hashes for a given prefix, I captured a short session of traffic where I visited a known currently blacklisted url "utfvq.portrelay.com" and firefox sent a request to google for full hashes and google responded with a list of hashes.
The prefixes firefox sent are(Hex encoded) : 2e2e372e,2e26382e,2e2e382e,6545382e
The 4 matching full hashes it received are :
2e26382e2e2e436d2e2e2e2e322e3b2e2e2e2e2e4a2e2e2e7b2e2e2e6a492e2e
6545382e2e2a5b792e652e2e2e2e2e2e2e2e70442e7d2e2e2e222e2e502e2e2e
2e2e382e6c36252e2e522e2e592e2e2e2e2e3f592e2e2e782e2e572e4e2e2e2e
2e2e372e2e2e2e2e55682e542e51622e552e2e68352e2e2e2e2e2e2e2ed2755
In my implementation however, the hash prefixes I generate do not seem to match the hash prefixes that firefox sent. Hence, I am not getting any full hash matches in my client.
I have followed googles description of the API closely and made sure the previous steps such as url canonicalization are implemented properly.
The url, SHA256 hashes I get are
utfvq.portrelay.com/ : 5c2383012676e63656c13167e1cc4f55309c4e1b73c22556e36ec1487e8b8697
portrelay.com/ : 842638fe92ee436da7808d0232d03bcaa0f5c8b64ad5eee97bf28dbb6a49f8ae
Can some one point out why the hashes do not match. I have followed the API guide to the best of my knowledge. Is there some implementation detail I am missing ?

It turned out to be a basic character encoding error on my part.
The SHA hashes I compute in my code are correct. The way I looked at the hashes that firefox sent was wrong. I copied the characters from a text file where it stored any byte not with in regular ascii range as a dot(.). Then I converted these to Hex values which is a kind of "lossy" encoding. This is why there were so many "2e" hex chars in the hashes. Now I am using just the original bytes and they match.

Related

how to Wrapping License Requests to drmtoday server via python

i'm trying to simulate drm media playing in python so i start with this demo from castlab "https://demo.castlabs.com/" and the video that i'm testing it is in MPEG-DASH format and drm system is widevine provided via cenc (if is that how to say it) , here is the url of the video
https://demo.cf.castlabs.com/media/msspr/Manifest?v=5.1.38
key id is :
8bdf77c0-5665-46c5-8ec8-422e571fc000
and the pssh is :
AAAAVHBzc2gAAAAA7e+LqXnWSs6jyCfc1R0h7QAAADQIARIQi993wFZlRsWOyEIuVx/AABoIY2FzdGxhYnMiEIvfd8BWZUbFjshCLlcfwAAqAlNE
schemeIdUri (widevine) is :
edef8ba9-79d6-4ace-a3c8-27dcd51d21ed
and the drm server url :
https://lic.staging.drmtoday.com/license-proxy-widevine/cenc/
the drm server need an custom header witch is :
dt-custom-data: eyJ1c2VySWQiOiJwdXJjaGFzZSIsInNlc3Npb25JZCI6InAwIiwibWVyY2hhbnQiOiJzaXgifQ==
and never change its base64 encoded from :
{"userId":"purchase","sessionId":"p0","merchant":"six"}
and it also need the body request or payload request , and thats where my problem is
i was able to catch the body request in arraybuffer from chrom developer tool and converted to base64 an this is what i find
CAESyR8STApKCjQIARIQi993wFZlRsWOyEIuVx/AABoIY2FzdGxhYnMiEIvfd8BWZUbFjshCLlcfwAAqAlNEEAEaEO9+id0bff5JFPIQdYcCqRoYASDtl+rzBTAVQu4eChJzdGFnaW5nLmdvb2dsZS5jb20SEChwNFTACPY2GK3nRD22xMgasByr1bNXgrfr+XFlzU+UcctZsvWfVXNmF9vKXmkIBD7/Ivlx6fq7KNVx/izq28XI3axu6FPqPgb4NUTURq0SeuJnYzLLPItXccgCeZrgpqBbXoxLy716wO10/mIMTyb4pIMNoPVHdnm08MD+vKIYY0Xm7ZQ4ZYSljcDolY8tePvDD+hsJSZ+BDLJiQ57PQHSsrjAH90F6N3TIK0RgEAXk+Sm9oe1qQa4W9QQQQq6SUsLqAL/mtdFDkK6m3m3Wn9ArUUwiytpsM+FPrsgmRkMnGKxLPtX9iQdQAcV0MxuVQs22d/OcI264AExZG4zG431RXzbfNHUVTqFNO3dSSyrtbNYZnZ4HqimAl1Cx0/KQQq4v3AL0ZAueDJzJA7ahb7c1jYGiYvgj+OkQBYsaiv/s9lmPdZkDZLAn8d4ghDL1TEmvOLglu7GhJz8WxSyFPiZtOx2CN2jnMTjEimfAtw2w70eToQGbrINt9FksB7BirxMUwMAO2vy8HCMytLj6WbqhzUKtFFQcSkQRdKdP7vVaSKL+LYribtva5MREwIVu59zr2kCzpIDlb+l/Z2B8SBsKWwRGIcLifB/WxwobPYKs9PcUEZ4KytSbnGqjOsydb9SWcCSzs1kryo/BWL59pWt6LBYb+6DwLvYAcq5vJDn5H9qLnEWZpuhVNzOPX0HFZ0tOtckPwJMCzv45ZB05ePrnedJxZCt4guxKlTczFgGkuRfU3VhFnbuQJep6xN7w/UMvDHA26VNeyRb7pn9HJrT8DgxYotfWBPgWbRbDqIWJqylA13FPeGUaNDrPWBeE5VsGqZNkuLf9VDXZ2FHBskFETRxOQZ46wnUG1x0YZYuOtjbAcgKKUO6IOEB/BhATU7iv4+VRxxq+PCBTffFsMVSxwjbc2LwqA0KTXugeFPPoUYfUSJ/T4XfOmms6LtSbG5R1XoQJlsi3rZtD6Gmcqyuul5ij1W8Whpj4oHNEpt3lUg2Kk9Czrz/xzsj+lZlUocoQ/438pBCUiRr0L88whivLYweb6gkidsqXJI+gbv1tba8utOb1abgHuhpc5l/RWQeXi1xZqBOJEhLdRljDK5PRJnTjfWGmHnpbAdYGEuJTAnWoIfIcC/sIsR1/epMgD9D4BzjZL5ls374FX1nGPS77Ont7Jc7xHu9Zv9G9LyFjM5RvEcBIoNXxsRbVlTkXkXTW7QM0bZktSS4DJC29Q2J5PxZD8X2ecBUhvOYEWqpHg4a+SM/5gbLJIK9HJWa64oV1XCtjn5yYuYqv8mfJpuuUu4QuGKYzkKMADZCdUqMJBYY7kWVHWtklZQPJnv2VDNcdbDRbc2xNrxQZTpTr4rXVuY5zSFeYQE6WVAiEjjCJMPMFKKMz7g2d+BgnufEM+hnpU5pgORBvDDvfy5y5E4qDmKgNsxFlYapTBzqZfgR411+4PjqhS0J3llsAjcc4ypCmSd7ywY76Bgx0YXpnWp50HoWJ2IYt95I5moYbV8SGTCEbfU0xZjL2bhm4fZDuXqzljfS/Z6/199uzp6m7bUoDkuqDAhV//HZxk8oxkw4iYx632U7naoEoT7aDmsHL+V6dr12P0hfYU6f0A081r56T2QPGeRta6tpENVeCLNvpiGF0EeeNfNiquVPa8V0jhgN9dRwnfepvQOjBbGWL+OfhB3wfJ/NggDrT5ZzN9L89UC+9XUW7mBAM/RdCS/vraBYbxPgbxGvCgaNVHYU47RUCIDUydo4U3By7aACr85FkEveVn/UEnNJKG9jP1VPNlSixAiACZyCRkkbwyUdfJwlYse7fszgO4BoNfsS4KAgVjzqZ3y9plCLbMBxzP4J4gJJ68Qb3tQolbR4I4hSch8lpdwT9Le3qipM/jJ55hNR9wvS+UAIE22lXOGFb8T+sOovHWSWPBwTZAiqdcfWlZSTodaUvjvXEIjIE1CidZWb3gdaYKmKoQIzP64W++ZkVKTm/dYm3mzbL+SK/d0dc5ZNWaglDs+TUv3jm1joYNQUcm9IAn/OHfHLNNpJq8nnnirOoGa/nNcKjVIcQSiXxMMC//CUtOLbJae1Fae1A1ghreuzXJhUCqEfwQkhKb8HW7S+i4UCk6ArVSbUrnK0Yx5h99K6zuqdj6yisDB/2kWw4agQYX0gDekXoMMj3x04HSNB0BqxSVYGz5HzLrn39xsW/UV1UzTfXvNpgTaSYjyh6HaO5JByJ2FcY3yg3doH5C/dAfyvkYPjRx6zlvM1saze3x/UJz6jjtRfACTgBcefxWGhap2DBsamJoIPitJMp7MyNTzOddzKdsiQwEQBRED8/JVosTZ+ojKDFw8d1lQC0HD93j7TW5DNJbJvttFGSf3op0k1WJth5l5PUXT2GkUEkCTFQBzuQehIRmj5arWwKbYWXcArGhTX/cr7DQYzczI2PTaBEWHrSJBIeUuPxpttZRT+Umt/fZykNA929JYg10jmEUICuJ5uMLiOn9n5ocpp+XU9RDxHk7QfgkEztsI0HC5CUIlnS8kTxBnjoh+vBXIv8ovdov9VeM8AuaPVS+41sgHHvInz7rvQlAGnhBa2zp00NSOCWabUbKu7/hD4vucll+gEE9i6yArkleVveog/22sU+9fkhY2+xozzNtYUfwPKj/Ylv0+inMzLgARiqCH7duZxNeecJvVIVVSqhLm84DCTfG0TzVRZl7ZwHHrZtk2LaqdvZQbx1oES4Ltq1nEwz04rGD/x6zPYd4NjU+IpNO6lDHZm5dxgwfcN+fu6Oz3ffBL30BwmaczSfQrAc9FZda/q0O5BKOVSCiwXHIf6Q/F7CZmtwkCzUSLoLXMo5CSRmoD0Zt3j7NDr26ckgpgcJqvjv7xXt+SndB1a1E21QphVSaYJD2ICxkZUqg4t5gpjQhxUKYKVkG1Eu/uhRPsLYdqn2eMG6Gfmcco8LvhhRFw7RpUSrZjHAkiPrS9E2xWtVOGKkRBwRaOUyGKz8h9nD0yXHVBkN9/VFlEzWV172hkjIsGopMu88ds8GOrJysVST+D6RCe9URLPsQpXUa8rELnrbzacdUcW7QyG8cC2YoruW4SrL96UWWwxqc3S0cbX1fNSNfpiaRysCOyQHyJwHz+ySAYx5HKIiavJ+sSq0oOkVzPbBECjsLlZ5cv6psQW96UG7U6Z1StnpQgsVzxS6wYwnxtBZ+b5GcHnmyqIKlWFUC7pWZo1tg/J0CZMNPXJ1xwD1yl8t+p3HoPhPqLa6qaBOZ0WO+4TM4WoXxv+VMyNJnCk51oU7O3DYKVunXWNpdysKJ9NkuBGqa8Qp8gp9tW1ojlNicXBMG1ExOnWTbrhSCp1idhlqXWOl867pkzupUJVe6ZMsy5wnxm0r7EG1aT30Kbeb7hXM9nKyH3r6FXn5hBF97oXWXnbxYHQ/p1TMRqQ8y27yuBUKZn8QjeZGBZsjX8N3XLv1mSKUrvyj9qrpBtepngVN9vmXMPF/RT6d2VHK7RIsu3kEkhFPaE18zRkE1ENcvSBSMUeCiy6hgCQG6Y0PK2earSoMMCSl2WpRqr2Pz1AvA5DYyn6UxCNbznvE8AcM1Qz0a4gtSEcvfTAUXtlq5L79H6YHVmK7P2vl6a+lRhha2SSCuBn9x8SvKpmBehylFjPvYDvE+nzkEh2nr+8vxHx/VdwLu7S2HyXqod07h+CLcUpvmyRGkyEs0aB51rlfczeyBaMgBwJiSw751GVSx53NlWbph0HrqmbPOpKQg9x/PrzksiJtYSEVTL6/Z81FmV2ThM1VcdEMcAfXqJNCho6IF0QAXSSEipTxcQJdvpswXe7gV9iOPRJs3WJSWB93i8QjzCZXGQpgj81BT8V3RI2ki1CLfaZTgQVfA9uST0SWalTZfWvUOVq7DScR02/HRtM1QkClIIuFa5vWlDXVoEjCV1hC74PHo7y8jmQ+/99G3Uov8HWA68yESIqRjkh/Y+aR+6RstUQf9x7o8Hst9QlxAvv/QmjlzzmxSZWyi8qg/fFYG2WoqdMELkpViZjJqI6u/TvJx26/tY/4q68Lc2wgCCZbBGeJJftBGa7Sc0CyWAMatGgBCYw5NX1CjDE/WwfiuZG0Xj4Jb+2xGl2lgmqK5SXWEGdYgTtkmSYHF1aTzthNYRJwwQmnxo+DKwMsDIXAkot7U6dRilWAUPzUalve4G85R4qkbQIHi5qWavO3fQAfCGS0SCx0dPzcdQ1+hp8FXS/kE7zx/xUDPaQqa/Mp/RHL1QpadtWbdjLwnltWmzAaUVG+w/kSk9o5BmZyeKizZ7JdswWPPbqJWj+3bhzss//dBJJILXCKwrUL0vSQlWfaxiCp1dtmHnaSbtgAI6pQETemrPGMmxoZMg4XdJ9+pYWDGd38r+Hwh2femihulnxDsUqRo6vkyNKX+thEgnXFS2AFpRCHaUSgRbCKnJMA8VtaYBOgG9rZu++T6po3ROrLT5tDyyvy+cRDRM3BTjYezCSTt4iE1xw98SSPjrdigkpAtokUS3+Jmf/yQq7Q+K2AvOLeQOkkEhynZs009rpAoJctycXGksMgeje+ViaU2ghyNANqDttGFc2tLWEatSAWoIy3dTKfuaTuTBrjAvVf6Dbl1ZcK3FVfjxM1LprDT52hTuLhO5hb12sXWfg0nqd/m7OB7giBncDH3B/4qTMP8saCR3p3WKf8M0bEoDYLRnKHULbke7n+B4GzPM4iwqSxT1WJaydKgdQdsD3zmLXwTVTXD+k669JtxP0hxFT9jK+NAOOaUV/3ITCGzO5H2CetI534549s8HY+CYLZdofc4Ozzi8u3C3ix8AA+7oc73+uNhr8Nj7Z6jTsrlvOXoevW7UVVB0CzVufK9gfnzAodK+CO2tWzyIQLn7uNn6/Yd4XwCkDQPR4qCqAAiZNMFcg8nC83o6ZkZC5ofOoVkHs+zwpHS2fQ/dBj8BqJqAxjmnOnKS+vX3rZ8u4C/6VRIa3sqhMjbnQgEVO6NJ4kKKBk/9saU34eBXkEKLH3Trcph9KUxyt7FG0XcPc6QcmSrtPPu17Az7ght5GpAhk02DSq3s6jAKx3C6JlunHEr/HhBclL83n8BNia76ArZ7Hs291kd68sysS5edaw4plPyN9hL9wgBc1D79wFVC9yYlgP7z/XOdSiF4N1z35G5nl6UxBvtdqYTLYCw2+LG3Vhznf7GdCel5keCoM/g5doWyi2FqRyTshYdcfk2jseYJ96tC3BE64UwT20WlSYwwagAKSEYJ37jGNCidW/AJHF4H3gKxhNVWSAqun5aY2ZJum9y6R20zlDr269U496Fplcb1wEWgLWBJVgd1yVpbhPR8JbaIPMenOsPO4h/XlKHq/LxsqFL9J/X5V1kwjerBkFdHneb93HCe1nTdyczNMwPymkm1LxinLMG1eJ7Y8YDlWWlFVXQJN7hIH2TUuozxDZTnC9XFfSRdNIjagWu2fujKrZjV+KjqD7VztIocL5LaYLnxPobQsSNGDqTih6qwU6EhHv90e2e1BT5G0yW7RIMYfe83SfW1lyoXWxq/YZieapVolSTY1rQ6zNVcsG0GmBDzm2ZVVKS5PCOuZiW6t4q+T
i really don't know how this code was formed i know that it converted to arraybuffer "Unit8 and "int8" and used as body request to the server , i coped this base64 code and try to decoded in this site "https://tools.axinom.com/decoders/LicenseRequest" and it give me information about the license request like Certificate and Serial Number and the signature plus the pssh data and content id , algorithm (AES-CTR)
now since most of the information that are stored in that base64 code are in my hand (pssh,kid,system id), i want to know how to create\build\wrapper the request in the same way that base64 code was build to use it in python
The request is generated by interacting with the CDM in the browser via the EME API: https://w3c.github.io/encrypted-media/. You can learn more by using this: https://chrome.google.com/webstore/detail/eme-call-and-event-logger/cniohcjecdcdhgmlofniddfoeokbpbpb
Maybe this helps to call Chromium from Python: Python - how to load Google Chrome or Chromium browser in the gtk.Window like webkit.WebView()?
You would also need to take care to enable Widevine in CEF, and there might be limitations associated with that because of some signature verification process. See
https://bitbucket.org/chromiumembedded/cef/src/master/include/cef_web_plugin.h#lines-168
and look for "VMP" here: https://github.com/castlabs/electron-releases

Verifying digital signatures in PDF documents

I'm trying to verify PDF's digital signatures.
I know that when a PDF is signed, a byterange is defined, the certificates get embedded, and from what i've read, the signed message digest and the timestamp are also stored in the PDF.
I already can extract the certificates and validate them.
Now I'm trying to validate the pdf's integrity and my problem is I don't know where the signed message digest is located.
In this sample signed pdf from Adobe (http://blogs.adobe.com/security/SampleSignedPDFDocument.pdf), i can clearly identify the digest since it is down below the embedded certificates: /DigestMethod/MD5/DigestValue/ (line 1520).
But that PDF sample seems to be from 2009, and I suspect the message digest is stored in a different way now, because I signed a PDF with Adobe Reader and also with iText, and I can't find any message digest field like the previous one.
Can someone tell if the digests are now stored in a different way? Where are they located?
Anyway, for now I'm using that sample document from Adobe, and trying to verify its integrity.
I'm getting the document's bytes to be signed acording to the specified byterange, and digesting them with MD5 algorithm, but the digest value I get doesn't match with the one from the message digest field...
Am I doing something wrong? Is the digest also signed with the signer's private key?
I appreciate any help.
There are numerous details to get right when calculating the hash for integrated PDF signatures, among them:
Extract the correct bytes from the PDF to hash. The ByteRange tells you exactly which byte ranges are signed. To be accepted in modern signing contexts, the ranges must cover the whole PDF file revision with the exception of the value of Contents.
Beware, the value of Contents includes the the leading '<' and the trailing '>' brackets.
Don't use a regular text editor or text processing instructions (like readln or writeln) to process PDFs. PDFs are binary in nature, even if they look textual to the naked eye. Copying PDF parts using such text related operations most likely changes them in details, definitively breaking the signature hash value.
When in doubt, don't guess but read the specification. A copy of ISO 32000-1 has been made available by Adobe here, and much you need to know about the PDF format to start processing them can be found there and in other public standards referenced in there. A very short introduction to integrated PDF signatures can be found in this answer and documents referenced from there.

OAuth 1.0 binary data upload with RESTful api

I have a DESKTOP application which interacts with a web service through their RESTful api and OAuth 1.0. I can use all the resources fine, authentication, get/post calls to retrieve and send simple text data etc, no problems there.
However, I'm struggling to send binary data. The service allows sending pictures. For one part, they have a non-OAuth api through which you can also send a file in plain binary, simply pass some params in the URL and put the entire binary file in the post.
Now to do this through the OAuth api, becomes an issue:
The service specifies the post needs 2 parameters (not to be included in the URL since this is REST, but in the POST parameters):
image A binary file, base64 data, or a URL
type The type of file that's being sent in. Accepted values: file, base64, url
So we have 2 parameters: image contains the file itself, and type would be "file" to specify binary. But how am I supposed to include this through OAuth? given that the -image- and -type- parameters (and their values) must be used to generate the base string for the signature, which is matched against the parameters included in the POST.. I can't include the entire binary there as a value for the image parameter; so how is it done in this case?
nvm, their OAuth api expects the file as multipart/form-data with the variables there, and none in the base string for the signature (their non-OAuth api instead doesn't need multipart, so I expected their OAuth wouldn't either; their documentation is a bit lacking in this area).

What is Twitter's definition of an URL?

I already asked the same question over at dev.twitter.com, however, I didn't get an answer there. So maybe someone here on SO ran into the same issue and has an answer.
In my application I count the length of the characters the user enters to compose a tweet. However, if the user enters an URL, this will be shortened automatically (by Twitter's API) when posting the tweet. So I have to replace the length of the URL with the length of the resulting t.co URL in my character counter.
However, the problem is now, what is Twitter's definition of an URL so that I know when you adapt my character counter and when not. For example www.verylongexampleurl.de gets shortened, while verylongexampleurl.de (without the www) doesn't, but verylongexampleurl.com does get shortened again.
I couldn't find any documentation, but maybe I missed it. All hints are appreciated.
Quoting from dev.twitter.com:
Need help parsing tweet text?
Take a look on the Twitter text processing library we’re using for auto linking and extraction of usernames, lists & hashtags.
Ruby: https://github.com/twitter/twitter-text-rb
Java: https://github.com/twitter/twitter-text-java
Javascript: https://github.com/twitter/twitter-text-js
The actual specification (tests) can be found here: https://github.com/twitter/twitter-text-conformance/blob/master/autolink.yml

Twitter API Strange Encoded Chars Returned

I'm building a little app that uses the twitter search api: http://search.twitter.com/search.json?q=funny
Now everything is working great, but sometimes it messes up my PHP script when the tweets contain chineze chars. Now on my site I use UTF-8 meta headers. Can someone tell me how to convert the following chars which were outputted by the Twitter API to a readable format?
EXAMPLE OUTPUT:
\u525B\u624D\u5728igfw.tk\u770B\u5230\u6709\u4E00\u500B\u535A\u5BA2\u63D0\u4F9B\u6BCF\u534A\u5C0F\u6642\u6539\u5BC6\u78BC\u7684ssh,\u62FF\u4F86\u7DF4\u7FD2\u4E00\u4E0Bbash\u8173\u672C\uFF0C\u65B7\u958B\u5F8C\u6703\u81EA\u52D5\u91CD\u65B0\u9023\u63A5\uFF0C\u4F7F\u7528Ctrl-C\u9000\u51FA,Cygwin\u4E0A\u7DE8\u5BEB\uFF0CLinux\u8A72\u6C92\u554F\u984C\uFF0CMAC\u81EA\u884C\u6E2C\
the \uXXXX are unicode codes for the characters
you could use a table of values pairing with the utf. or pack the string as json and use json_decode that already has that behavior.
you can see an example of how to do the conversion in the Service_json.json_decode() source, as one implementation that comes to my mind now
http://pear.php.net/package/Services_JSON/redirected