I have a Google sheets formula that fetches data from an API:
function GETDATA(data){
var res = UrlFetchApp.fetch("https://some.api.com/" + data);
var content = res.getContentText();
return content;
}
While everything works fine, this script fires an API call each time it's used in a cell (=GETDATA("cars") for example). This can quickly result in hundreds of API calls, at which the API gives up and responds with a code 429: too many requests.
Is there a way, that I can slow these requests down in Google App Script? I already tried
function GETDATA(data){
Utilities.sleep(150);
var res = UrlFetchApp.fetch("https://some.api.com/" + data);
var content = res.getContentText();
return content;
}
but this just results in all the cells waiting 150 ms and then executing all together again.
Related
I have written the code:
function getId(username) {
var infoUrl = "https://www.instagram.com/web/search/topsearch/?context=user&count=0&query=" + username
return parseInt(fetch(infoUrl)['users']);
}
function fetch(url) {
var ignoreError = {
"muteHttpExceptions": true
};
var source = UrlFetchApp.fetch(url, ignoreError).getContentText();
var data = console.log(source);
return data;
}
To get the userID of the username input.
The error corresponds to the line:
return parseInt(fetch(infoUrl)['users']);
I have tried differnt things but I cant get it to work. The url leads to a page looking like this:
{"users": [{"position": 0, "user": {"pk": "44173477683", "username": "mykindofrock", "full_n........
Where the numbers 44173477683 after the "pk": are what I am trying to get as an output.
I hope someone can help as I am very out of my depth, but I guess this is how we learn! :)
I was surprised that the endpoint you provided actually led to a JSON file. I would have thought that to access the Instagram API, you would need register a developer account with Facebook etc. Nevertheless, it does return a JSON by visiting in the browser. I suppose that it just shows the publicly available information on each user.
However, with Apps Script it seems like a different story. I visited:
https://www.instagram.com/web/search/topsearch/?context=user&count=0&query=user
In a browser and chose a random user id. Then I called it from Apps Script with UrlFetchApp:
function test(){
var username = "username7890543216"
var infoUrl = "https://www.instagram.com/web/search/topsearch/?context=user&count=0&query=" + username
var options = {
'muteHttpExceptions': true
}
var result = UrlFetchApp.fetch(infoUrl, options)
console.log(result.getResponseCode())
}
Which returns a 429 response. Which is a "Too Many Requests" response. So if I had to guess, I would say that all requests to this unauthenticated endpoint from Apps Script have been blocked. This is why when replacing the console.log(result.getResponseCode()) with console.log(result.getContentText()), you get a load of HTML (not JSON) part of it which says:
<title>
Page Not Found • Instagram
</title>
Though maybe its IP based. Try and run this code from your end, unless you get a response code of 200, it is likely that you simply can't access this information from Apps Script.
You are setting data to the return value of console.log(source) which is undefined. So no matter what the data is, you will get undefined.
Another thing to avoid is that fetch will not necessarily be hoisted because fetch is a built in function to make API calls.
I have long-running ASP.NET Core API that I would like to present a progress bar on UI.
From the server-side, I know how many jobs will be done from the very beginning. Say, if I have 10 jobs and each job takes a second, this will be 10 seconds long progress bar.
The best I could find was https://github.com/DmitrySikorsky/AspNetCoreUploadingProgress, But it relies on saving the progress on Startup. Progress, which is static int. Wouldn't it mean there can be only one upload session at a time in the entire web application?
I wonder if I can do this with axios call:
return axios.post(
'/api/long-running-task',
{},
{
onUploadProgress: function(progressEvent) {
console.log("upload", progressEvent.loaded, progressEvent.total);
},
onDownloadProgress: function(progressEvent) {
console.log("download", progressEvent.loaded, progressEvent.total);
}
}
);
And if I do something properly from ASP.NET Core side, would I be able to communicate back to axios and trigger either onUploadProgress or onDownloadProgress?
What I tried:
[HttpPost]
[Route("long-running-task")]
public async Task<ActionResult> Run()
{
Response.StatusCode = 200;
Response.ContentType = "text/plain";
Response.ContentLength = 10;
var sw = new StreamWriter(Response.Body);
for (var i = 0; i < 10; i++)
{
await Task.Delay(1000);
await sw.WriteAsync("1");
await sw.FlushAsync();
}
return null;
}
axios writes one upload log shortly after, and then one download log 10 seconds later. No interim progress is received.
I found a way to make this work: specify ContentType to be text/event-stream.
I believe it changes server side caching behavior so it should work on any browsers as along as axios is supported. Confirmed working on Chrome 81, Edge 44 and IE 11.
[HttpPost]
[Route("long-running-task")]
public async Task<ActionResult> Run()
{
Response.StatusCode = 200;
Response.ContentType = "text/event-stream";
Response.ContentLength = 10;
var sw = new StreamWriter(Response.Body);
for (var i = 0; i < 10; i++)
{
await Task.Delay(1000);
await sw.WriteAsync("1");
await sw.FlushAsync();
}
return Ok();
}
EDIT: at the end, return Ok() instead of null. If you don't have any jobs, returning null will throw an exception.
Upload and download status wont help you with your task since it will take progress of request size.
What you need to do is on asp.net core side:
Create job/start api which returns you some kind of key
job/progress/id api which will tell you how much you progress
Then inside of your axios call start job get id, then each N seconds call status to see how much your job progressed.
Other option look into signalR steps will be the same the only you wont need to call status api every N seconds you will be able to push back from server you job status.
I want to ask about increase performance when i do multiple future http request in single page. In case , i want to build a dashboard page. In dashboard, i've 4 endpoints url that return different result in every endpoint and should be shown in dashboard page.
here example code when load data
var client = new http.Client();
Future main() async {
var newProducts = await client.get("${endpoint}/product?type=newly&limit=5");
ProductListResponse newProductResponse = ProductListResponse.fromJson(json.decode(newProducts.body));
var bestSeller = await client.get("${endpoint}/product?type=best-seller&limit=5");
ProductListResponse bestSellerResponse = ProductListResponse.fromJson(json.decode(bestSeller.body));
var outOfStock = await client.get("${endpoint}/product?type=out-of-stock&limit=5");
ProductListResponse outOfStockResponse = ProductListResponse.fromJson(json.decode(outOfStock.body));
var lastRequest = await client.get("${endpoint}/product-request?type=newly&limit=5");
ProductRequestListResponse productRequestResponse = ProductRequestListResponse.fromJson(json.decode(lastRequest.body));
}
every endpoint when i hit manually using postman it takes 200ms for return the result. But when i implement in flutter app, it took almost 2s.
can i improve performance when getting data?
The reason why your code run so slow is that you are making those HTTP requests one by one. Each await will take quite some time.
You can either not use await and implement the logic using callbacks (.then) or you can combine the Futures into one using Future.wait and use await for that combined Future.
Your code will look something like this:
var responses = await Future.wait([
client.get("${endpoint}/product?type=newly&limit=5"),
client.get("${endpoint}/product?type=best-seller&limit=5"),
client.get("${endpoint}/product?type=out-of-stock&limit=5"),
client.get("${endpoint}/product-request?type=newly&limit=5")
]);
I am using .net core 2.2 for my flight listing application and i am using wego api for that. but while i am using the below code for getting flights from wego api i am not getting the complete response, but in postman i am getting full result set at one request.
public async Task<SearchResultMv> GetFlights(FlightParam flightParam, AuthResult auth)
{
var request = new HttpRequestMessage(HttpMethod.Get, "https://srv.wego.com/metasearch/flights/searches/" + flightParam.SearchId + "/results?offset=0&locale=" + flightParam.locale + "¤cyCode=" + flightParam.currencyCode);
request.Headers.Add("Bearer", auth.access_token);
request.Headers.Add("Accept", "application/json");
var client = _httpClient.CreateClient();
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", auth.access_token);
var response = await client.SendAsync(request).ConfigureAwait(false);
SearchResultMv json = new SearchResultMv();
response.EnsureSuccessStatusCode();
if (response.IsSuccessStatusCode)
{
json = await response.Content.ReadAsAsync<SearchResultMv>().ConfigureAwait(false);
return json;
}
}
Some time I am not getting any result set by the above code. Wego api is not providing any pagination or filtration on this api. so Please help me to achieve this. Thanks for advance.
According the their documentation you need to poll their API to get the results gradually. You also need to increment the offset as the results are returned.
For example, if the first set of results gives you 100 results, the following request should have the offset value set as 100. offset=100.
Documentation:
https://developers.wego.com/affiliates/guides/flights
Edit - Added sample solution
Sample code that polls the API every second until reaching the desired number of results. This code hasn't been tested so you'll need to adjust it to your needs.
const int numberOfResultsToGet = 100;
var results = new List<SearchResultMv>();
while (results.Count < numberOfResultsToGet)
{
var response = await GetFlights(flightParam, auth);
results.AddRange(response.Results);
// update offset
flightParam.Offset += response.Results.Count;
// sleep for 1 second before sending another request
Thread.Sleep(TimeSpan.FromSeconds(1));
}
Change your request to use a dynamic Offset value. You can add the Offset property to the FlightParam class.
var request = new HttpRequestMessage(
HttpMethod.Get,
$"https://srv.wego.com/metasearch/flights/searches/{flightParam.SearchId}/results?" +
$"offset={flightParam.Offset}" +
$"&locale={flightParam.locale}" +
$"¤cyCode={flightParam.currencyCode}");
I'm having no luck getting a response from v4 of the Google Sheets API when running against a public (i.e. "Published To The Web" AND shared with "Anyone On The Web") spreadsheet.
The relevant documentation states:
"If the request doesn't require authorization (such as a request for public data), then the application must provide either the API key or an OAuth 2.0 token, or both—whatever option is most convenient for you."
And to provide the API key, the documentation states:
"After you have an API key, your application can append the query parameter key=yourAPIKey to all request URLs."
So, I should be able to get a response listing the sheets in a public spreadsheet at the following URL:
https://sheets.googleapis.com/v4/spreadsheets/{spreadsheetId}?key={myAPIkey}
(with, obviously, the id and key supplied in the path and query string respectively)
However, when I do this, I get an HTTP 401 response:
{
error: {
code: 401,
message: "The request does not have valid authentication credentials.",
status: "UNAUTHENTICATED"
}
}
Can anyone else get this to work against a public workbook? If not, can anyone monitoring this thread from the Google side either comment or provide a working sample?
I managed to get this working. Even I was frustrated at first. And, this is not a bug. Here's how I did it:
First, enable these in your GDC to get rid of authentication errors.
-Google Apps Script Execution API
-Google Sheets API
Note: Make sure the Google account you used in GDC must be the same account you're using in Spreadsheet project else you might get a "The API Key and the authentication credential are from different projects" error message.
Go to https://developers.google.com/oauthplayground where you will acquire authorization tokens.
On Step 1, choose Google Sheets API v4 and choose https://www.googleapis.com/auth/spreadsheets scope so you have bot read and write permissions.
Click the Authorize APIs button. Allow the authentication and you'll proceed to Step 2.
On Step 2, click Exchange authorization code for tokens button. After that, proceed to Step 3.
On Step 3, time to paste your URL request. Since default server method is GET proceed and click Send the request button.
Note: Make sure your URL requests are the ones indicated in the Spreadsheetv4 docs.
Here's my sample URL request:
https://sheets.googleapis.com/v4/spreadsheets/SPREADSHEET_ID?includeGridData=false
I got a HTTP/1.1 200 OK and it displayed my requested data. This goes for all Spreadsheetv4 server-side processes.
Hope this helps.
We recently fixed this and it should now be working. Sorry for the troubles, please try again.
The document must be shared to "Anyone with the link" or "Public on the web". (Note: the publishing settings from "File -> Publish to the web" are irrelevant, unlike in the v3 API.)
This is not a solution of the problem but I think this is a good way to achieve the goal. On site http://embedded-lab.com/blog/post-data-google-sheets-using-esp8266/ I found how to update spreadsheet using Google Apps Script. This is an example with GET method. I will try to show you POST method with JSON format.
How to POST:
Create Google Spreadsheet, in the tab Tools > Script Editor paste following script. Modify the script by entering the appropriate spreadsheet ID and Sheet tab name (Line 27 and 28 in the script).
function doPost(e)
{
var success = false;
if (e != null)
{
var JSON_RawContent = e.postData.contents;
var PersonalData = JSON.parse(JSON_RawContent);
success = SaveData(
PersonalData.Name,
PersonalData.Age,
PersonalData.Phone
);
}
// Return plain text Output
return ContentService.createTextOutput("Data saved: " + success);
}
function SaveData(Name, Age, Phone)
{
try
{
var dateTime = new Date();
// Paste the URL of the Google Sheets starting from https thru /edit
// For e.g.: https://docs.google.com/---YOUR SPREADSHEET ID---/edit
var MyPersonalMatrix = SpreadsheetApp.openByUrl("https://docs.google.com/spreadsheets/d/---YOUR SPREADSHEET ID---/edit");
var MyBasicPersonalData = MyPersonalMatrix.getSheetByName("BasicPersonalData");
// Get last edited row
var row = MyBasicPersonalData.getLastRow() + 1;
MyBasicPersonalData.getRange("A" + row).setValue(Name);
MyBasicPersonalData.getRange("B" + row).setValue(Age);
MyBasicPersonalData.getRange("C" + row).setValue(Phone);
return true;
}
catch(error)
{
return false;
}
}
Now save the script and go to tab Publish > Deploy as Web App.
Execute the app as: Me xyz#gmail.com,
Who has access to the app: Anyone, even anonymous
Then to test you can use Postman app.
Or using UWP:
private async void Button_Click(object sender, RoutedEventArgs e)
{
using (HttpClient httpClient = new HttpClient())
{
httpClient.BaseAddress = new Uri(#"https://script.google.com/");
httpClient.DefaultRequestHeaders.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json"));
httpClient.DefaultRequestHeaders.AcceptEncoding.Add(new System.Net.Http.Headers.StringWithQualityHeaderValue("utf-8"));
string endpoint = #"/macros/s/---YOUR SCRIPT ID---/exec";
try
{
PersonalData personalData = new PersonalData();
personalData.Name = "Jarek";
personalData.Age = "34";
personalData.Phone = "111 222 333";
HttpContent httpContent = new StringContent(JsonConvert.SerializeObject(personalData), Encoding.UTF8, "application/json");
HttpResponseMessage httpResponseMessage = await httpClient.PostAsync(endpoint, httpContent);
if (httpResponseMessage.IsSuccessStatusCode)
{
string jsonResponse = await httpResponseMessage.Content.ReadAsStringAsync();
//do something with json response here
}
}
catch (Exception ex)
{
}
}
}
public class PersonalData
{
public string Name;
public string Age;
public string Phone;
}
To above code NuGet Newtonsoft.Json is required.
Result:
If your feed is public and you are using api key, make sure you are throwing a http GET request.In case of POST request, you will receive this error.
I faced same.
Getting data using
Method: spreadsheets.getByDataFilter has POST request