I would like to reverse engineer an API Call of an android app.
It's a POST request, octet-stream.
The issue is that the content is binary but mixed with clear text.
Here's a little snippet of the start of an API call content (got it using MITM proxy):
'\x12\xf3\x01\
x09\xb4\xda\xc3^(mH#\
x11y\xe9&1\x08L\
x02#\x1a\
x05helix2$30 Rue N\xc3\xa9laton, 75015 Paris, FranceJ\x0f30 Rue N\xc3\xa9latonR\x02enZ\x0f30 Rue N\xc3\xa9latonb\
As you can see, clear text is mixed with hex code, and all those calls contents always start with \x12. I would like to understand how this app generates this api call. I decompile it using jadx, but it was just too complicated with the thousands of classes and obfuscation.
Is there any way to do this without going through the actual apk code? I'm afraid that this is just some type of encoding and that you don't need all the hassle of reverse engineering the actual app code
Thank you
Related
I'm using Flask-Restplus to marshal responses in a Flask server. I also began using this package to parse HTTP requests from the client, when I stumbled about this huge warning on their site:
Warning
The whole request parser part of Flask-RESTPlus is slated for removal
and will be replaced by documentation on how to integrate with other
packages that do the input/output stuff better (such as marshmallow).
I then switched to Marshmallow to validate/parse the HTTP client requests. Thus, the workflow in my server is:
Client request (HTTP GET/POST) ...
--> process request with Marshmallow and validate/format data
--> do stuff with DB (read, update, create)
--> Format output response with Flask RESTPLUS
... Client response
So far this works well. However, is this the correct way to use Flask RESTPLUS and Marshmallow along? On the marshmallow website, there is no clear direction towards a specific use of this package. The documentation just says:
Marshmallow is an ORM/ODM/framework-agnostic library for converting
complex datatypes, such as objects, to and from native Python
datatypes.
I have seen examples in the web where people use Marshmallow to format the output response, and Flask to validate expected data (with #api.expect). Which approach is better?
Also, I wonder if this even makes sense to use Flask RESTPLUS at all? It seems the only interest of this library is to have the Swagger UI doc automatically generated. Other than that, Marshmallow can do everything that Flask RESTPLUS does. So maybe I missed out something, can anyone help or comment?
Thanks
Take further notice of the warning displayed on the website:
Don’t worry, if you have code using that now and wish to continue doing so, it’s not going to go away any time too soon.
The developers will post documentation how to integrate best Marshmellow in the future.
So far this works well. However, is this the correct way to use Flask RESTPLUS and Marshmallow along? On the marshmallow website, there is no clear direction towards a specific use of this package
Flask is a microframework: what this means for you is that much of the implementation of your application is up to the programmer because it lacks most of the functionality which is common to expect in a full-fledged web application framework (eg. Django, Pyramid et al.) and there's not "one way and only one way" of doing things in it. Implementation details like data validation are up to you to provide via plugins, libraries or even implementing them yourself (not recommended).
I wonder if this even makes sense to use Flask RESTPLUS at all?
From what I've seen in the Quick start page of Flask-RESTPlus, it provides useful models that facilitates exposing REST verbs for resources, endpoints, arg parsing and data formatting. But again, this question depends mostly on you and your application requirements.
I'm rewording a question I asked earlier in hopes that it will get me some feedback.
I need to generate a static map from Mapbox that shows two end points and the polyline route between them. The method outlined below is what I'm using. It sometimes works and sometimes doesn't--and that is what's driving me crazy.
First, I use the two GPS end points to make an API request that generates the polyline. Then I use a second API request with the two end points and the polyline to generate the static map. The first part always seems to work. I get an encoded polyline that I can run through an online decoding tool and it shows the various points along the path just fine, but the second step to generate the map from Mapbox sometimes gives me the expected map and sometimes gives me an error code. The code is usually {"message":"Not Found"}, but sometimes I get a {"message":"Not Authorized - No Token"} error code.
I use this API with the two end points to get a JSON file that has a polyline:
https://api.mapbox.com/directions/v5/mapbox/driving/POINT1;POINT2?access_token=TOKEN
Then, once I have the polyline, I am using this API to generate the map:
https://api.mapbox.com/styles/v1/mapbox/streets-v11/static/pin-s-a+9ed4bd(POINT1),pin-s-b+000(POINT2),path-5+f44-0.5(POLYLINE)/auto/500x300?access_token=TOKEN
This works fine...
Point 1 = -87.321536,36.584454
Point 2 = -104.793676,38.749513
Polyline = }kh~Ez{}sO|HUuHmj#uhHbOyAla#kisAdelC|aMf~w#egO|uaAahfBr{|#w_#sAozkCcwFsp#v]__w#hjFjf#bdCkaJhmzAiyXtzb#fJ|{#{p[|~yErtBdacE{x^vkvHrZfuKo[tpBfxSrcuBocGpi_AqhCvjb#wgAvfsDhyc#fkrD~mD~bkDayD~ywCsui#vjmEoyBpk|AiOhHrvEdueApda#huoA|v[zuhFsySvwaAvbDwqHxXpwDYj#riJULheEfXh^tdAj~I^gIhAkEY~#kHseBlGyC^kA
Map API request = https://api.mapbox.com/styles/v1/mapbox/streets-v11/static/pin-s-a+9ed4bd(-87.321536,36.584454),pin-s-b+000(-104.793676,38.749513),path-5+f44-0.5(}kh~Ez{}sO|HUuHmj#uhHbOyAla#kisAdelC|aMf~w#egO|uaAahfBr{|#w_#sAozkCcwFsp#v]__w#`hjFjf#bdCkaJhmzAiyXtzb#fJ|{#{p[|~yErtBdacE{x^vkvHrZfuKo[tpBfxSrcuBocGpi_AqhCvjb#wgAvfsDhyc#fkrD~mD~bkDayD~ywCsui#vjmEoyBpk|AiOhHrvEdueApda#huoA|v[zuhFsySvwaAvbD`wqHxXpwDYj#riJULheEfXh^tdAj~I`^gI`hAkEY~#kHseBlGyC^kA)/auto/500x300?access_token=TOKEN
Map (yay!)
This does not work...
Point 1 = -87.321536,36.584454
Point 2 = -85.855218,37.690038
Polyline = }kh~Ez{}sO|HUtCurC}xCkhC}~BqvIuc_#o~l#oaI}aUrs#kkL}~Iso\qHoyGsiKaTpQcRdlGusK_DEiz#w^cwOakJurAizT{hCugF_bBenSsOclZkdCw]mgEwyDogLgfBatEubCmkGxa#icLe_Do_Spf#cw]}sEgOaDgT{[zFyIk~Bts#
Map API request = https://api.mapbox.com/styles/v1/mapbox/streets-v11/static/pin-s-a+9ed4bd(-87.321536,36.584454),pin-s-b+000(-85.855218,37.690038),path-5+f44-0.5(}kh~Ez{}sO|HUtCurC}xCkhC}~BqvIuc_#o~l#oaI}aUrs#kkL}~Iso\qHoyGsiKa`TpQcRdlGusK_DEiz#w^cwOakJurAizT{hCugF_bBenSs`OclZkdCw]mgEwyDogLgfBatEubCmkGxa#icLe_Do_Spf#cw]}sEgOaDgT{[zFyIk~Bts#)/auto/500x300?access_token=TOKEN
No Map :-(
This is a small part of a much bigger project where I am automating the process for generating project reports, but the problem isn't with my jQuery/AJAX code. It's a problem with my APIs and/or Mapbox. If there is a better/faster/easier way to get these maps, then I am absolutely open to suggestions, because this does not seem like the most efficient way to do it. I will be generating these reports from a computer. I am not worried about bandwidth or scaling down for a phone application. The file can be as cumbersome as it wants to be as long as I can consistently generate a map every time.
Steve's comment hints at the right solution, but just to make it absolutely clear - this is happening because you are not properly encoding the path polyline in your request. Polylines in particular have a nasty habit of including reserved characters that will break URIs unless properly encoded at the time of the request.
When I encode the polyline portion of your path overlay with a simple tool like url encode decode, I'm able to make your request work as expected:
https://api.mapbox.com/styles/v1/mapbox/streets-v11/static/pin-s-a+9ed4bd(-87.321536,36.584454),pin-s-b+000(-85.855218,37.690038),path-5+f44-0.5(%7Dkh~Ez%7B%7DsO%7CHUtCurC%7DxCkhC%7D~BqvIuc_%40o~l%40oaI%7DaUrs%40kkL%7D~Iso%2FqHoyGsiKa%60TpQcRdlGusK_DEiz%40w%5EcwOakJurAizT%7BhCugF_bBenSs%60OclZkdCw%5DmgEwyDogLgfBatEubCmkGxa%40icLe_Do_Spf%40cw%5D%7DsEgOaDgT%7B%5BzFyIk~Bts%40)/auto/500x300?access_token=MAPBOX_ACCESS_TOKEN
⚠️ disclaimer: I currently work at Mapbox ⚠️
I'm using SoapUI to automate tests against my companies APIs. I've successfully setup and run dozens of these cases.
This tyme, I'm getting an error which, after exhaustive tracking down, I've found is due to our APIs requiring the "#" char itself rather than the URL friendly "%40" substitution.
The post request I want is structured like this:
https://<endpoint>.com/<resource>?<param>&email_address#example.com
And what I'm seeing made is:
https://<endpoint>.com/<resource>?<param>&email_address%40example.com
How can I enforce SoapUI to use the char itself?
I've tried setting headers, media type and representations (though possibly not through all permutations).
Thanks.
Use Disable Encoding for the parameter.
I've been creating Presigned HTTP PUT URLs and everything was working great until I wanted to start using "folders" in S3; I wanted the key to have the character '/'.
Now I get Signature doesn't match when I send the HTTP PUT requests due to the fact the '/' probably changes to %2F... If I escape the character before creating the presigned URL it works great, but then the Amazon console management doesn't understand it and shows it as one file instead of subfolders.
Any idea?
P.s.
The HTTP PUT requests are sent using C++ with POCO NET library.
EDIT
I'm using Poco HttpRequest from C++ to my Java web server to generate a signed url (returned on the response).
C++ then uses this url to put a file in s3 using Poco again.
The problem was that the urls returned from the web server were parsed through Poco URI objects that auto decoded the s3 object key thus changing it.With that in mind I was able to fix my problem.
Tricky - I'll try to approach this bottom up.
Disclaimer: I got carried away visually inspecting the Poco libraries instead of actually debugging a code sample, which should yield more reliable results much faster, see below ;)
Analysis
If I escape the character before creating the presigned URL it works
great, but then the Amazon console management doesn't understand it
and shows it as one file instead of subfolders.
The latter stems from S3 not having a concept of folders on the storage level actually, see e.g. section Index Documents and Folders within Index Document Support:
Objects stored in Amazon S3 are stored within a flat container, i.e.,
an Amazon S3 bucket, and it does not provide any hierarchical
organization, similar to a file system's. However, you can create a
logical hierarchy using object key names and use these names to infer
logical folders that contain these objects.
That's exactly what the AWS Management Console is doing here as well:
The AWS Management Console also supports the concept of folders, by
using the same key naming convention used in the preceding sample.
However, your test regarding the assumption of / being encoded as %2F proves, that this is indeed how Poco::Net is encoding the URL when performing the HTTP PUT request.
(I'm actually a bit surprised that the AWS Java SDK seems to generate different URLs here for / vs. %2F, insofar a recent analysis regarding Why is my S3 pre-signed request invalid when I set a response header override that contains a “+”? seems to indicate respective canonicalization by the AWS .NET SDK, see below for more on this.)
Potential Solution
In order for your scenario to work as desired, you'll need to figure out where the URL is encoded this way - I could think of two components in principle:
Poco::Net
Finding out why Poco::Net is encoding the URL different than S3 (if at all, see below) is best done by debugging your code, here's where I'd start:
Class HTTPRequest uses class URI in turn, which automatically performs a few normalizations on all URIs and URI parts passed to it, in particular percent-encoded characters are decoded. The other way round is handled by method encode(), which is where things get interesting and call for a breakpoint, see URI.cpp:
lines 575 ff. - here encode() does its magic, which indeed seems to be in place, insofar neither the code within the function nor the various chars passed in via the reserved parameter contain the offending / (see lines 47 ff. for the respective constants in use)
consequently you might want to set a breakpoint in this function and backtrace the callstack to find out which code is actually doing the encoding upfront, which might not yield an offender at all, see below.
Java => C++ transition
You haven't specified yet, which channel is actually used to communicate the pre-signed URL generated by the AWS Java SDK to C++ in turn. Given the code review (mind you, visual inspection only, I haven't debugged this myself yet) of the Poco::Net functionality yields the conclusion, that no obvious offender can be identified in the library itself, thus it seems more likely that it might already enter your C++ layer encoded (easily verified via debugging of course) - are you by chance using any kind of web service between these components for example?
Good luck!
I have a really weird WCF problem here...
We're connecting to a crappy third-party web service; it was a nightmare to even get it going, we had to create a custom WCF binding since those guys decided to use "ISO-8859-1" as their text encoding (instead of UTF-8 like everyone else on the web), and the other settings were messy, too - and not documented anywhere, of course...
It's been working ok for a while now, but suddenly, some of our data coming back in mangled up. We expect to get back names of places, and being in Switzerland, some of those have German umlauts in them. But for the past two or three months, we suddenly get back
Hünibach
instead of the proper
Hünibach
So the ü (u umlaut) is mangled.
No problem, I figured they finally switched to UTF-8, and I changed my custom binding to use UTF-8 as its text encoder instead of ISO-8859-1 - but no luck - no I'm getting:
EXCEPTION: System.ServiceModel.Security.MessageSecurityException
The HTTP request was forbidden with client authentication scheme 'Basic'.
What the f????? The service is protected by a username/password which we pass in using the ClientCredentials of WCF. Seems that changing the text encoding somehow messes up the credentials !?!?! Weird.....
OK - back to ISO-8859-1, and I just tried to interpret the response payload as UTF-8 - again no luck :-( Tried with UTF-16, UTF-32, UTF-7 even, Unicode, BigEndianUnicode - all to no avail.
So how on earth do I get back my proper umlauts, and still be able to call that bloody service... works just fine in SoapUI, btw.....
Any ideas?? I'm desperately grasping at any straws you might throw me!!
Try inspecting the data you are getting back and see what numeric codes they are using to represent it. Umlaut is one of those characters in 8859-1 that shares code with other characters.
See second para in - http://en.wikipedia.org/wiki/%C3%9C#Typography
Actually, I finally figured out what the trouble was.
For some reason, changing the sample CustomTextEncoder (provided by Microsoft in the WCF & WF samples) to use UTF-8 instead of ISO-8859-1 doesn't work.
On the other hand, ripping out the custom text encoder from my custom binding and just using the standard TextMessageEncoder that WCF provides from the get go (which uses UTF-8 by default) did work.
Don't ask me why.... that's just the facts I found.....