I am completely new to coding. I am trying to build a dashboard in Klipfolio. I am using a CATSone API to pull data from CATSone to Klipfolio. However, I can only get 100 rows a time, which means I would have to pull data 2600 times.
I am now trying to build a script to get data from the API through Google Script Editor. However, since I have no experience in this, I am just trying stuff. I watched some videos, also from Ben Collins. The basis is simple, and I get what he is doing.
However, I have a problem with putting the API key.
var API_KEY = 'key'
function callCATSone(){
//Call the CATSone API for all candidate list
var response = UrlFetchApp.fetch("https://api.catsone.nl/v3/candidates");
Logger.log(response.getContentText());
// URL and params for the API
var url = 'https://api.catsone.nl/v3/candidates';
var params = {
'method': 'GET',
'muteHttpExceptions': true,
'headers': {
'Authorization': 'key ' + apikey
}
};
// call the API
var response = UrlFetchApp.fetch(url, params);
var data = response.getContentText();
var json = JSON.parse(data);
}
In the end, I would like to transfer all candidate list data to my sheets. Therefore, I call on the API with Authorization key. After that, I will manipulate the data, but that's for later. The first problem I now encounter, is this fail code:
'Verzoek voor https://api.catsone.nl/v3/candidates is mislukt. Foutcode: 401. Ingekorte serverreactie: {"message":"Invalid credentials."} (Gebruik de optie muteHttpExceptions om de volledige reactie te onderzoeken.) (regel 6, bestand 'Code')'.
I expect to get a list of all data from CATSone into my sheets.
Does anyone know how I can accomplish this?
Two changes should fix the credentials error:
Authorization header should be Authorization: 'Token ' + yourApiKey instead of 'key ', see the v3 API documentation https://docs.catsone.com/api/v3/#authentication.
API key in your case is stored in a global variable API_KEY, you should reference it exactly like that, not as an apikey (unless there is a typo in your sample or some missing code): Authorization : 'Token ' + API_KEY.
Btw, it should probably set either a Content-Type header or a contentType parameter for UrlFetchApp.fetch() method call to application/json as UrlFetchApp.fetch() request content type defaults to application/x-www-form-urlencoded.
If you plan to continue working with APIs, it would be beneficial to read this MDN article.
Related
I am testing the Coinbase API endpoints with Postman and the challenge is when I need to paginate
In order to setup Postman, I have followed the guide available here and in summary:
added variables
coinbase-api-base
coinbase-api-key
coinbase-api-secret
coinbase-api-timestamp
coinbase-api-signature
Added pre-request script in order to generate the request signature
// 1. Import crypto-js library
var CryptoJS = require("crypto-js");
// 2. Create the JSON request object var req = { timestamp: Math.floor(Date.now() / 1000), // seconds since Unix epoch method:
pm.request.method, path: pm.request.url.getPath(), body: '', // empty
for GET requests message: undefined, secret:
pm.collectionVariables.get("coinbase-api-secret"), // read value from
collection variable hmac: undefined, signature: undefined, };
// 3. Create the message to be signed req.message = req.timestamp + req.method + req.path + req.body;
// 4. Create HMAC using message and API secret req.hmac = CryptoJS.HmacSHA256(req.message, req.secret);
// 5. Obtain signature by converting HMAC to hexadecimal String req.signature = req.hmac.toString(CryptoJS.enc.Hex);
// 6. Log the request console.info("request: ", req);
// 7. Set Postman request's authentication headers for Coinbase REST API call pm.collectionVariables.set("coinbase-api-timestamp",
req.timestamp); pm.collectionVariables.set("coinbase-api-signature",
req.signature);
all worked well for a simple request such as:
GET {{coinbase-api-base}}/v2/accounts
then, if I add in the body request parameter (as explained here):
limit=50
to change the default pagination, I get an authentication error....
"errors": [
{ "id": "authentication_error",
"message": "invalid signature"
}
questions:
how can I fix it?
how the body of the request can play with the request signature...
any help suggestion is much appreciated
Thank you
Edit: the below being said, I'm not sure the base accounts API supports paging I could be wrong though, the CB docs are inconsistent to say the least. It does seem that the account history (ledger) and holds do though.
https://docs.cloud.coinbase.com/exchange/reference/exchangerestapi_getaccounts
get accounts function in Node.js API doesn't give an args param where the ledger does (see below):
getAccounts(callback) {
return this.get(['accounts'], callback);
}
Documentation for an api that does support paging, notice it gives you a query param section not available in the accounts documentation:
https://docs.cloud.coinbase.com/exchange/reference/exchangerestapi_getaccountledger
Looking at the node api, you still need to add the query string params to the body in order to sign:
calling function:
return this.get(
['accounts', accountID, 'ledger'],
{ qs: args },
callback
);
signing function:
let body = '';
if (options.body) {
body = JSON.stringify(options.body);
} else if (options.qs && Object.keys(options.qs).length !== 0) {
body = '?' + querystring.stringify(options.qs);
}
const what = timestamp + method.toUpperCase() + path + body;
const key = Buffer.from(auth.secret, 'base64');
const hmac = crypto.createHmac('sha256', key);
const signature = hmac.update(what).digest('base64');
return {
key: auth.key,
signature: signature,
timestamp: timestamp,
passphrase: auth.passphrase,
};
You can't add the limit to the body of the request, GET requests never includes any body.
You should add it as a query string parameter like (this is just an example):
GET {{coinbase-api-base}}/v2/accounts?limit=50
I have written the code:
function getId(username) {
var infoUrl = "https://www.instagram.com/web/search/topsearch/?context=user&count=0&query=" + username
return parseInt(fetch(infoUrl)['users']);
}
function fetch(url) {
var ignoreError = {
"muteHttpExceptions": true
};
var source = UrlFetchApp.fetch(url, ignoreError).getContentText();
var data = console.log(source);
return data;
}
To get the userID of the username input.
The error corresponds to the line:
return parseInt(fetch(infoUrl)['users']);
I have tried differnt things but I cant get it to work. The url leads to a page looking like this:
{"users": [{"position": 0, "user": {"pk": "44173477683", "username": "mykindofrock", "full_n........
Where the numbers 44173477683 after the "pk": are what I am trying to get as an output.
I hope someone can help as I am very out of my depth, but I guess this is how we learn! :)
I was surprised that the endpoint you provided actually led to a JSON file. I would have thought that to access the Instagram API, you would need register a developer account with Facebook etc. Nevertheless, it does return a JSON by visiting in the browser. I suppose that it just shows the publicly available information on each user.
However, with Apps Script it seems like a different story. I visited:
https://www.instagram.com/web/search/topsearch/?context=user&count=0&query=user
In a browser and chose a random user id. Then I called it from Apps Script with UrlFetchApp:
function test(){
var username = "username7890543216"
var infoUrl = "https://www.instagram.com/web/search/topsearch/?context=user&count=0&query=" + username
var options = {
'muteHttpExceptions': true
}
var result = UrlFetchApp.fetch(infoUrl, options)
console.log(result.getResponseCode())
}
Which returns a 429 response. Which is a "Too Many Requests" response. So if I had to guess, I would say that all requests to this unauthenticated endpoint from Apps Script have been blocked. This is why when replacing the console.log(result.getResponseCode()) with console.log(result.getContentText()), you get a load of HTML (not JSON) part of it which says:
<title>
Page Not Found • Instagram
</title>
Though maybe its IP based. Try and run this code from your end, unless you get a response code of 200, it is likely that you simply can't access this information from Apps Script.
You are setting data to the return value of console.log(source) which is undefined. So no matter what the data is, you will get undefined.
Another thing to avoid is that fetch will not necessarily be hoisted because fetch is a built in function to make API calls.
I'm trying to make an authenticated api call to VALR crypto exchange as first step towards automated trading. They provide most of the code so I thought it would be easy even as a non coding techie. The code below does actually create the correct HMAC SHA512 signature using the API Secret provided for testing but I have a problem in passing this result along to the next section of code to request balances (starting at line 17). If I cut and paste the result/displayed 'signature' and 'timestamp' (after running the code) back into the code it does in fact work. So what changes do I need to make the code automatically pick up the signature and timestamp. The user defined function appears to keep all parameters "secret" from the rest of the code, especially after using return.
import time
import hashlib
import hmac
def sign_request( api_key_secret,timestamp, verb,path,body=""):
payload = "{}{}{}{}".format(timestamp, verb.upper(), path, body)
message = bytearray(payload, 'utf-8')
signature = hmac.new(bytearray(api_key_secret, 'utf-8'), message, digestmod=hashlib.sha512).hexdigest()
print("Signature=",signature)
print ("Timestamp",timestamp)
return signature
sign_request( verb = "GET", timestamp = int(time.time()*1000),path="/v1/account/balances",api_key_secret="4961b74efac86b25cce8fbe4c9811c4c7a787b7a5996660afcc2e287ad864363" )
import requests
url = "https://api.valr.com/v1/account/balances"
payload = {}
headers = {
'X-VALR-API-KEY': '2589fb273e86aeee10bac1445232aa302feb37e27d32c1c599abc3757599139e',
'X-VALR-SIGNATURE': 'signature',
'X-VALR-TIMESTAMP': 'timestamp'
}
response = requests.request("GET", url, headers=headers, data = payload)
print(response.text.encode('utf8'))
Well after some hard thinking I decided to change to using global variables. The hmac still worked fine and gave me a signature. Then I removed the quotes around signature and timestamp and realised they were both integers. I was then able to convert that signature and timestamp to a string and everything started to work perfectly. Maybe someone else will make use of this. If you want to make a POST request remember to put single quotes around anything in the {body} statement to make it a string.
Here is the code that I am currently using for a GET request from VALR. It's been working fine for many months. You will need to change the path and the url to correspond to whatever you are trying to get, and obviously you will need to add your_api_key and your_api_secret.
If you need to send through other request parameters like transaction types etc. then you will ned to include them in the path and the url e.g. https://api.valr.com/v1/account/transactionhistory?skip=0&limit=100&transactionTypes=MARKET_BUY¤cy=ZAR
def get_orders(): # to get open orders from valr
timestamp = int(time.time()*1000)
verb = "GET"
path = "/v1/orders/open"
body = ""
api_key_secret = 'your_api_secret'
payload = "{}{}{}".format(timestamp, verb.upper(), path)
message = bytearray(payload, 'utf-8')
signature = hmac.new(bytearray(api_key_secret, 'utf-8'), message, digestmod=hashlib.sha512).hexdigest()
timestamp_str = str(timestamp)
url = "https://api.valr.com/v1/orders/open"
headers = {
'Content-Type': 'application/json',
'X-VALR-API-KEY': 'your_api_key',
'X-VALR-SIGNATURE': signature,
'X-VALR-TIMESTAMP': timestamp_str,
}
response = requests.request("GET", url, headers=headers, data=body)
dict = json.loads(response.text)
dict = pd.DataFrame.from_dict(dict)
print(dict)
I created a Service Bus / Notification Hub in my Azure Portal.
Now I'm trying to use the Azure REST API with Postman based on this doc :
https://msdn.microsoft.com/en-us/library/azure/dn223265.aspx
Here is the Postman configuration I have :
It's a POST method of the following url (Create Registration)
https://mysite.servicebus.windows.net/mysite-notif/registrations/?api-version=2015-01
(I replaced with mysite in that url for privacy reasons)
In the Headers, I typed 2 entries :
Content-Type
application/atom+xml;type=entry;charset=utf-8
Authorization
Endpoint=sb://[mysite].servicebus.windows.net/;SharedAccessKeyName=DefaultFullSharedAccessSignature;SharedAccessKey=[mykey]
(this Connection information I copied from the Azure portal)
In the Body, I chose raw - XML(txt/xml) and pasted :
<?xml version="1.0" encoding="utf-8"?>
<entry xmlns="http://www.w3.org/2005/Atom">
<content type="application/xml">
<WindowsRegistrationDescription xmlns:i="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://schemas.microsoft.com/netservices/2010/10/servicebus/connect">
<Tags>myTag, myOtherTag</Tags>
<ChannelUri>{ChannelUri}</ChannelUri>
</WindowsRegistrationDescription>
</content>
</entry>
(it's the Native registration for Windows Notification Service example)
When I send this call from within Postman, I get a 401 Error :
<Error>
<Code>401</Code>
<Detail>MalformedToken: The credentials contained in the authorization header are not in the WRAP format..TrackingId:ee0d87ef-6175-46a1-9b35-6c31eed6049d_G2,TimeStamp:8/13/2015 9:58:26 AM</Detail>
</Error>
What am I missing ?
Is it the Authorization tab I left on "No Auth" in Postman ?
Is it the value of the Authorization header that should be encoded like shown here ?
Creating registration ID for Azure Notification Hub via REST api
Thanks.
Here is an example of a pre-request script for postman that generates the needed header:
function getAuthHeader(resourceUri, keyName, key) {
var d = new Date();
var sinceEpoch = Math.round(d.getTime() / 1000);
var expiry = (sinceEpoch + 3600);
var stringToSign = encodeURIComponent(resourceUri) + '\n' + expiry;
var hash = CryptoJS.HmacSHA256(stringToSign, key);
var hashInBase64 = CryptoJS.enc.Base64.stringify(hash);
var sasToken = 'SharedAccessSignature sr=' + encodeURIComponent(resourceUri) + '&sig=' + encodeURIComponent(hashInBase64) + '&se=' + expiry + '&skn=' + keyName;
return sasToken;
}
postman.setEnvironmentVariable('azure-authorization', getAuthHeader(request['url'], "mySharedAccessKeyName", "mySharedAccessKey"));
postman.setEnvironmentVariable('current-date',new Date().toUTCString());
To use it do the following:
add this pre-request script to your postman request
replace mySharedAccessKeyName , mySharedAccessKey with your credentials
add a header Authorization: {{azure-authorization}}
add a header x-ms-date: {{current-date}}
Your "Authorization" header is not correct.
As stated in the Azure Notification Hubs REST API documentation, e.g. for creating a registration, the "Authorization" header has to contain the "Token generated as specified in Shared Access Signature Authentication with Service Bus"...
The token format is specified in the documentation for Shared Access Signature Authentication with Service Bus as the following:
SharedAccessSignature sig=<signature-string>&se=<expiry>&skn=<keyName>&sr=<URL-encoded-resourceURI>
URL-encoded-resourceURI: The url you send the POST request to (in your case "https://mysite.servicebus.windows.net/mysite-notif/registrations/?api-version=2015-01")
keyName: In your case the default key name "DefaultFullSharedAccessSignature"
expiry: The expiry is represented as the number of seconds since the epoch 00:00:00 UTC on 1 January 1970.
signature-string: The signature for the SAS token is computed using the HMAC-SHA256 of a string-to-sign with the PrimaryKey property of an authorization rule. The string-to-sign consists of a resource URI and an expiry, formatted as follows:
StringToSign = <resourceURI> + "\n" + expiry;
resourceURI should be the same as URL-encoded-resourceURI (also URL encoded)
Compute the HMAC-SHA256 of StringToSign using the SAS key (what you replaces with [mykey] in your example). Use the URL encoded result for signature-string then.
After spending over an hour trying to understand why the steps above didn't work, I realized if you are using the code from https://code.msdn.microsoft.com/Shared-Access-Signature-0a88adf8 It has two things that are not defined at the top of the code. Key and KeyName.
The Key is the part that alluded me because at first glance on the other post here I thought it was the same. Its not.
In Azure: Go to your Notification Hub, Then Click > Settings> Access Policies then on the Policy that has Manage Permission. Add a policy if you need to. Once you Click on the Access Policy. It shows Connection String, Primary and Secondary. Copy the Primary to your Clipboard and throw it in notepad. It will look something like this..
Endpoint=sb://mysite.servicebus.windows.net/;SharedAccessKeyName=DefaultFullSharedAccessSignature;SharedAccessKey=hc7qZ+pMG6zltjmASDFrskZO+Yv52D55KQUxUTSO0og=
SharedAccessKeyName = KeyName
SharedAccessKey = Key
Yea it looks obvious all spelled out here but you cannot see this information in AZURE portal unless you copy it.
So Just to be totally Clear, in the header you generate the key "sig" by combining + "\n" + expiry which Baris did point out, but then you sign it with the Key not the KeyName..
I may sounds like an idiot spelling this out but this process is not an easy one.
Hope it helps someone else.
Baris Akar's response is mostly correct, except for one omission that, for whatever reason, is also not mentioned in the relevant documentation: the signature parameter (i.e., the signature-string in sig=) must be Base64 encoded!
You have to remove "\"" in Token String like below.
authorizationString = resultA.replaceAll("\"","");
From
"SharedAccessSignature sr=https%3a%2f%2fmshub.servicebus.windows.net%2f&sig=PFZVab43PMsO0q9gz4%2bFsuaQq%5ff05L4M7hKVBN8DEn0%3d&se=1553339810&skn=RootManageSharedAccessKey"
To
SharedAccessSignature sr=https%3a%2f%2fmshub.servicebus.windows.net%2f&sig=PFZVab43PMsO0q9gz4%2bFsuaQq%5ff05L4M7hKVBN8DEn0%3d&se=1553339810&skn=RootManageSharedAccessKey
Good luck.
See the following documentation from Microsoft to generate a SAS Token.
This token you can use in Postman.
Generate SAS Token (NodeJs, Java, etc.)
Like Jérôme, I also used the example at https://code.msdn.microsoft.com/Shared-Access-Signature-0a88adf8 to generate the token and I also found out that the .NET-generated token worked. I compared the .NET-generated token with my ruby-generated token and found that URI.escape did not encode the last character (an '=' sign) of my base64 hash. It also did not encode '+' signs. Adding the string '=+' to the function fixed the problem: URI.escape(hmacb64, '=+')
(I don't know if there are other characters that should be identified here.)
It also took me quite some time to figure out a way to generate the SAS tokens in Go.
I created a gist which shows how to generate those tokens:
https://gist.github.com/dennis-tra/14c63e6359f17cbb504e78d6740ca465
I probably wouldn't have figured it out if had not found this repo:
https://github.com/shanepeckham/GenerateSASTokenGo/blob/master/gosas.go
Working from D-rk's code, which is probably outdated in 2022, here's an updated version that works in Postman 10.5.6 and with the Azure Notification Hub's api-version 2020-06
Postman Pre-Request Script:
function createSharedAccessToken(sb_name, eh_name, saName, saKey) {
if (!sb_name || !eh_name || !saName || !saKey) {
throw "Missing required parameter";
}
var resourceUri = encodeURIComponent("https://" + sb_name + ".servicebus.windows.net/" + eh_name)
// Set expiration in seconds
var expires = (Date.now() / 1000) + 20 * 60;
expires = Math.ceil(expires);
var toSign = CryptoJS.enc.Utf8.parse(resourceUri + '\n' + expires);
var sa_key_utf8 = CryptoJS.enc.Utf8.parse(saKey);
var hmac = CryptoJS.HmacSHA256(toSign, sa_key_utf8);
var hmacBase64 = CryptoJS.enc.Base64.stringify(hmac);
var hmacUriEncoded = encodeURIComponent(hmacBase64);
// Construct autorization string
var token = "SharedAccessSignature sr=" + resourceUri + "&sig=" + hmacUriEncoded + "&se=" + expires + "&skn="+ saName;
return token;
}
var sb_name = "your-notification-hub-namespace";
var eh_name = "your-notification-hub-name";
//See Access Policies -> Connection String
var sa_name = "your-shared-access-key-name"
var sa_key = "your-shared-access-key-name"
var auth_header = createSharedAccessToken(sb_name, eh_name, sa_name,sa_key);
pm.environment.set('azure-authorization',auth_header);
pm.environment.set('current-date',new Date().toUTCString());
Solution provided by Dirk helped me to resolve the issue.
But make sure to use SharedAccessKeyName and SharedAccessKey from a policy which has "Manage" claims access. If you have only Send and/or Listen claims, then the authentication will not work and throws an error - MalformedToken: The credentials contained in the authorization header are not in the WRAP format
I am trying to use UlrFetch to submit CSV data to Zoho reports. I am using the following code:
function doImport(tabla,file) {
var url="https://reportsapi.zoho.com/api/xxxxxxxx/yyyyyyyyyyy/"+tabla;
var ticket="zzzzzzzzzzzzzzzz" ;//getTicket();
url=url + "?ZOHO_ACTION=IMPORT&ZOHO_OUTPUT_FORMAT=XML&ZOHO_ERROR_FORMAT=json&ZOHO_API_VERSION=1.0"
var params={"ZOHO_API_KEY":"vvvvvvvvvvvvvvvvvvvvvv"
,"ticket":ticket
,"ZOHO_FILE":file
,"ZOHO_IMPORT_TYPE":"APPEND"
,"ZOHO_ON_IMPORT_ERROR":"ABORT"
,"ZOHO_AUTO_IDENTIFY":"true"
,"ZOHO_CREATE_TABLE":"false"
,"ZOHO_DATE_FORMAT":"dd-MM-YYYY"
,"ZOHO_DELIMITER":"0"
};
var options =
{
"method" : "post",
"payload" : params,
"contentType": "multipart/form-data"
};
var response=UrlFetchApp.fetch(url, options);
var tableDataString=response.getContentText();
expireTicket(ticket);
Logger.log(tableDataString);
return tableDataString;
}
However, the data is not submitted in correct multiform format (getting error 500 status). This issue backtracks to early 2011. Please, one or two examples of how to submit blob files in multipart/form-data format would be welcome.
Thanks
For payload, you are passing it as an Object, which looks correct. This will be interpreted as an HTTP form (which you want).
To fix your script, try the following:
Make sure the value you're using for ZOHO_FILE is a Blob. This makes sure the HTTP form will automatically be sent with:Content-Type: multipart/form-data; boundary=[automatically determined]
Do not specify contentType for the HTTP POST. This allows UrlFetchApp to automatically use its own contentType value, which includes the boundary field. (Minor detail: It's ok to still specify contentType on the Blob itself, just not the overall post request. This allows specifying the contentType of each Blob within the post, if that interests you.)
UrlFetchApp will use multipart/form-data encoding automatically if you pass a Blob as a payload value. You may need to use:
"ZOHO_FILE": file.getBlob()