I am trying to send a text via get request to my server, but I have some struggles with spaces/special characters.
How can I encode my text for the request?
(And how to decode it in php?)
Here is my code:
var app = Application.currentApplication();
app.includeStandardAdditions = true;
var text = app.displayDialog("enter your text:", { defaultAnswer: "" }).textReturned;
var result = JSON.parse(app.doShellScript('curl https://example.com?text=' + text));
result
Well, it looks like I found the answer myself:
encodeURIComponent(text)
seems to be working for me. PHP decoding not necessary.
Related
I'm trying to make an authenticated api call to VALR crypto exchange as first step towards automated trading. They provide most of the code so I thought it would be easy even as a non coding techie. The code below does actually create the correct HMAC SHA512 signature using the API Secret provided for testing but I have a problem in passing this result along to the next section of code to request balances (starting at line 17). If I cut and paste the result/displayed 'signature' and 'timestamp' (after running the code) back into the code it does in fact work. So what changes do I need to make the code automatically pick up the signature and timestamp. The user defined function appears to keep all parameters "secret" from the rest of the code, especially after using return.
import time
import hashlib
import hmac
def sign_request( api_key_secret,timestamp, verb,path,body=""):
payload = "{}{}{}{}".format(timestamp, verb.upper(), path, body)
message = bytearray(payload, 'utf-8')
signature = hmac.new(bytearray(api_key_secret, 'utf-8'), message, digestmod=hashlib.sha512).hexdigest()
print("Signature=",signature)
print ("Timestamp",timestamp)
return signature
sign_request( verb = "GET", timestamp = int(time.time()*1000),path="/v1/account/balances",api_key_secret="4961b74efac86b25cce8fbe4c9811c4c7a787b7a5996660afcc2e287ad864363" )
import requests
url = "https://api.valr.com/v1/account/balances"
payload = {}
headers = {
'X-VALR-API-KEY': '2589fb273e86aeee10bac1445232aa302feb37e27d32c1c599abc3757599139e',
'X-VALR-SIGNATURE': 'signature',
'X-VALR-TIMESTAMP': 'timestamp'
}
response = requests.request("GET", url, headers=headers, data = payload)
print(response.text.encode('utf8'))
Well after some hard thinking I decided to change to using global variables. The hmac still worked fine and gave me a signature. Then I removed the quotes around signature and timestamp and realised they were both integers. I was then able to convert that signature and timestamp to a string and everything started to work perfectly. Maybe someone else will make use of this. If you want to make a POST request remember to put single quotes around anything in the {body} statement to make it a string.
Here is the code that I am currently using for a GET request from VALR. It's been working fine for many months. You will need to change the path and the url to correspond to whatever you are trying to get, and obviously you will need to add your_api_key and your_api_secret.
If you need to send through other request parameters like transaction types etc. then you will ned to include them in the path and the url e.g. https://api.valr.com/v1/account/transactionhistory?skip=0&limit=100&transactionTypes=MARKET_BUY¤cy=ZAR
def get_orders(): # to get open orders from valr
timestamp = int(time.time()*1000)
verb = "GET"
path = "/v1/orders/open"
body = ""
api_key_secret = 'your_api_secret'
payload = "{}{}{}".format(timestamp, verb.upper(), path)
message = bytearray(payload, 'utf-8')
signature = hmac.new(bytearray(api_key_secret, 'utf-8'), message, digestmod=hashlib.sha512).hexdigest()
timestamp_str = str(timestamp)
url = "https://api.valr.com/v1/orders/open"
headers = {
'Content-Type': 'application/json',
'X-VALR-API-KEY': 'your_api_key',
'X-VALR-SIGNATURE': signature,
'X-VALR-TIMESTAMP': timestamp_str,
}
response = requests.request("GET", url, headers=headers, data=body)
dict = json.loads(response.text)
dict = pd.DataFrame.from_dict(dict)
print(dict)
I'm trying to set up an API that generate PDF from web page (provided as URL). The API is gotenberg from thecodingmachine. I have it on Docker, it works just fine, I can't generate PDF through http request send with curl (for now I'm just trying to make it work, so I use the request provided as example in the documentation)
Now I am trying to make it work with my groovy/grails app. So I'm using the java tools to make the request.
Now here is my problem : the PDF file I get is blank (my app opend directly in my browser). It do has the right content, if I open it with the text editor, it's not empty, and it has almost the same content as the one I make using the curl request (which isn't blank).
I am 99% sure the problem come from the encoding. I tried changing the InputStreamReader encoding parameter, but it doesn't change anything. Here I put "X-MACROMAN" because that the encoding inside the pdf file that isn't blank, but it still doesn't change.
Here is my code :
static def execute(def apiURL)
{
def httpClient = HttpClients.createDefault()
// Request parameters and other properties.
def request = new HttpPost(apiURL)
MultipartEntityBuilder builder = MultipartEntityBuilder.create()
builder.addTextBody("remoteURL", 'https://google.com')
builder.addTextBody("marginTop", '0')
builder.addTextBody("marginBottom", '0')
builder.addTextBody("marginLeft", '0')
builder.addTextBody("marginRight", '0')
HttpEntity multipart = builder.build()
request.setEntity(multipart)
def response = httpClient.execute(request)
BufferedReader rd = new BufferedReader(
new InputStreamReader(response.getEntity().getContent(), "X-MACROMAN"))
StringBuffer result = new StringBuffer()
String line = ""
Boolean a = Boolean.FALSE
while ((line = rd.readLine()) != null) {
if(!a){
a = Boolean.TRUE
}
else {
result.append("\n")
}
result.append(line)
}
return result
I am 99% sure the problem come from the encoding. I tried changing the InputStreamReader encoding parameter, but it doesn't change anything. Here I put "X-MACROMAN" because that the encoding inside the pdf file that isn't blank, but it still doesn't change.
Did I made myself clear ? And does those who understands has any ideas why my PDFs are blank ?
I have a question about this create/Update leads API, http://developers.marketo.com/documentation/rest/createupdate-leads/.
There is no sample code for C# or JAVA. Only ruby available. So I have to try it by myself. But I always get null return from the response.
Here is my code:
private async Task<CreateLeadResponseResult> CreateLead(string token)
{
string url = String.Format(marketoInstanceAddress+"/rest/v1/leads.json?access_token={0}", token);
var fullUri = new Uri(url, UriKind.Absolute);
CreateLeadResponseResult createLeadResponse = new CreateLeadResponseResult();
CreateLeadInput input = new CreateLeadInput { email = "123#123.com", lastName = "Lee", firstName = "testtesttest", postCode = "00000" };
CreateLeadInput input2 = new CreateLeadInput { email = "321#gagaga.com", lastName = "Lio", firstName = "ttttttt", postCode = "00000" };
List<CreateLeadInput> inputList = new List<CreateLeadInput>();
inputList.Add(input);
inputList.Add(input2);
CreateLeadRequest createLeadRequest = new CreateLeadRequest() { input = inputList };
JavaScriptSerializer createJsonString = new JavaScriptSerializer();
string inputJsonString = createJsonString.Serialize(createLeadRequest);
using (var client = new HttpClient())
{
HttpResponseMessage response = await client.PostAsJsonAsync(fullUri.OriginalString, inputJsonString).ConfigureAwait(false);
// I can see the JSON string is in the message body in debugging mode.
if (response.IsSuccessStatusCode)
{
createLeadResponse = await response.Content.ReadAsAsync<CreateLeadResponseResult>();
}
else
{
if (response.StatusCode == HttpStatusCode.Forbidden)
throw new AuthenticationException("Invalid username/password combination.");
else
throw new ApplicationException("Not able to get token");
}
}
return createLeadResponse;}
//get null here.
Thank you.
-C.
The best way to debug this is to capture the exact URL, parameters and JSON that are submitted by your app and try submitting those manually via a tool like Postman (Chrome plug-in) or SOAP UI. Then you see the exact error message, which you can look up here: http://developers.marketo.com/documentation/rest/error-codes/. Based on that you can update your code. I don't know much about Java, but this is how I got my Python code to work.
Your example code was really helpful in getting my own implementation off the ground. Thanks!
After playing with it for a bit, I realized that the JavaScriptSerializer step is unnecessary since PostAsJsonAsync automatically serializes whatever object you pass to it. The double serialization prevents Marketo's API from processing the input.
Also, I agree with Jep that Postman is super helpful. But in the case of this error, Postman was working fine (using the contents of inputJsonString) but my C# code still didn't work properly. So I temporarily modified the code to return a dynamic object instead of a CreateLeadResponseResult. In debugging mode this allowed me to see fields that were discarded because they didn't fit the CreateLeadResponseResult type, which led me to the solution above.
I'd like to implement a FileUpload using the new FileReader API. From the client side, everything works well and I can send a PUT request to the server with the correct fields containing the file in Base64 encoded.
But in the server side, it's not going great, here are my results :
Logger.info(String.valueOf(request().body().asRaw())); // null
Logger.info(String.valueOf(request().body().asText())); // null
And most importantly :
Logger.info(String.valueOf(request().body().isMaxSizeExceeded())); // true !
What am I missing? How can I make it work?
I found the answer to my question !
For those who are looking for it, here's the answer :
You need to add a BodyParser as annotation for your method, and specify a higher maxLength value.
#BodyParser.Of(value = BodyParser.Json.class, maxLength = 1024 * 1024)
public static Result method() {
Logger.info(String.valueOf(request().body().asJson())); // Will not be empty!
}
I am new to Web Crawling, and I am using HttpWebRequest to crawl data from sites.
As of now I was successfully able to crawl and get data from my wordpress site. This data was a simple user profile data. (like name, email, AIM id etc...)
Now as an exercise I want to crawl wikipedia, where I will search using the value entered into textbox at my end and then crawl wikipedia with the search value and get the appropriate title(s) from the search.
Now I have the following doubts/difficulties.
Firstly, is this even possible ? I have heard that wiki has robot.txt setup to block this. Though I have heard this only from a friend and hence not sure.
I am using the same procedure I used earlier, but I am not getting the required results.
Thanks !
Update :
After some explanation and help from #svick, I tried the below code, but still not able to get any value (see last line of code, there I am expecting an html markup of the search result page)
string searchUrl = "http://en.wikipedia.org/w/index.php?search=Wikipedia&title=Special%3ASearch";
var postData = new StringBuilder();
postData.Append("search=" + model.Query);
postData.Append("&");
postData.Append("title" + "Special:Search");
byte[] data2 = Crawler.GetEncodedData(postData.ToString());
var webRequest = (HttpWebRequest)WebRequest.Create(searchUrl);
webRequest.Method = "POST";
webRequest.UserAgent = "Crawling HW (http://yassershaikh.com/contact-me/)";
webRequest.AllowAutoRedirect = false;
ServicePointManager.Expect100Continue = false;
Stream requestStream = webRequest.GetRequestStream();
requestStream.Write(data2, 0, data2.Length);
requestStream.Close();
var responseCsv = (HttpWebResponse)webRequest.GetResponse();
Stream response = responseCsv.GetResponseStream();
// Todo Parsing
var streamReader = new StreamReader(response);
string val = streamReader.ReadToEnd();
// val is empty !! <-- this is my problem !
and here is my GetEncodedData method defination.
public static byte[] GetEncodedData(string postData)
{
var encoding = new ASCIIEncoding();
byte[] data = encoding.GetBytes(postData);
return data;
}
Pls help me on this.
You probably don't need to use HttpWebRequest. Using WebClient (or HttpClient if you're on .Net 4.5) will be much easier for you.
robots.txt doesn't actually block anything. If something doesn't support it (and .Net doesn't support it), it can access anything.
Wikipedia does block requests that don't have their User-Agent header set. And you should use an informative User-Agent string with your contact information.
A better way to access Wikipedia is to use its API, rather than scraping. This way, you will get an answer that's specifically meant to be read by a custom applications, formatted as XML or JSON. There are also dumps containing all information from Wikipedia available for download.
EDIT: The problem with your newly posted code is that your query returns a 302 Moved Temporarily response to the searched article, if it exists. Either remove the line that forbids AllowAutoRedirect, or add &fulltext=Search to your query, which will mean you won't get redirected.