I can only read from the first 100,000 rows of any particular tab in a Google Sheet via the API.
Is this a known limitation of the Google Sheets API? I didn't see a reference to it in the documentation.
Issue and workaround:
I thought that if you want to retrieve the values from Spreadsheet, it seems that when the method of "spreadsheets.get" is used for the Spreadsheet including the large data, the data cannot be correctly retrieved because of an error like Response Code: 413. Message: response too large.. I thought that this might be the reason for your issue. And, in this situation, I confirmed that even when the method is changed from "spreadsheets.get" to "spreadsheets.values.get " and "spreadsheets.values.batchGet", the issue occurred. So I thought that this situation might be the current specification of Sheets API.
But, fortunately, I confirmed that when I tested to retrieve the values from your Spreadsheet using the query language, all values can be retrieved. And also, I confirmed that when I tested to retrieve the values from your Spreadsheet using the Spreadsheet service of Google Apps Script, all values can be retrieved.
So in this answer, I would like to propose these 2 patterns.
Pattern 1:
In this pattern, the values are retrieved by the query language. I thought that in your situation, this might be suitable. The endpoint is as follows.
https://docs.google.com/spreadsheets/d/###spreadsheetId###/gviz/tq?tqx=out:csv&gid=###sheetId###&access_token=###accessToken###
In this case, the access token can be also included in the request header instead of the query parameter. At that time, please use Authorization: Bearer ###. When the above endpoint is accessed using your access token, all values are returned as CSV data.
Pattern 2:
In this pattern, the values are retrieved by the Spreadsheet service of Google Apps Script. The sample script is as follows. When you use this script, please test this script at the script editor of Google Apps Script.
function myFunction() {
const id = "###spreadsheetId###";
const sheet = SpreadsheetApp.openById(id).getSheetByName("Tab 1");
const values = sheet.getDataRange().getValues();
console.log(values.length)
}
When I tested this script for your sample Spreadsheet, 405028 can be seen in the log.
References:
Query Language
Spreadsheet Service
Related
I'm trying to automate a process at my company where I have to send files to my customers every month.
These files are unique and each customer must receive their own. In addition, I wanted the client to be required to answer an NPS assessment form in order to have access to his file.
Honestly, I don't understand much about programming and I'm just trying to find a solution to do this automatically.
The solution I have in mind today is to use a google forms form and after filling out the form a link would be made available to access a google drive folder (that way the customer would be obliged to answer the form to have access to his file).
In the google drive, each file would be associated with a unique access key (for example, an identification number of that customer, in which he already knows what it is).
Then these customers should insert this key in the google drive folder in order to be able to access their file. That way I would be able to: get all the answers of the evaluation in a single form, make a standard message (or tutorial) available to all customers informing them how to access their file, and give access to the files automatically because the access key would be the same every month.
Sorry for the long text but I tried to describe my problem in the best way. I am accepting any kind of help or guidance. Even where I could get help about it. This automation would be very important to me because in addition to automating a process that costs a lot of time it would give me a very powerful tool to evaluate my services. Thanks in advance to everyone who took the time to read this question.
The best approach in this situation is to start by using Apps Script.
Apps Script is a powerful development platform which can be used to build web apps and specifically for your situation, automate tasks. What makes it special is the fact that it is easy to use and to create applications that integrate with G Suite.
Since you mention that the files are unique for each customer, what you can do is to store these values in a key - value list. However, depending on the number of the customers, you can store these value in a spreadsheet, something similar to this:
Making the files available only after submitting the Google Form mentioned can be done by taking advantage of Apps Script's triggers.
So essentially, you can use an onFormSubmit trigger which will execute every time the above mentioned form is submitted.
Code
You can also add another column in your spreadsheet to mark if the email was sent or not; taking this into account, this is how your spreadsheet will look like:
As for the code, this is just a short snippet which does the following:
opens the spreadsheet and retrieves the values for the File ID, Customer Email and Sent columns as the following arrays ids, emails and sent respectively;
retrieves the needed form and loops through the responses and checks if the last email of the form matches any of the ones from the spreadsheet;
if the conditions check, the file permissions are edited, an email is being sent, and the corresponding row on column C is being set to TRUE;
function editPermissions() {
let form = FormApp.openById('FORM_ID');
let sheet = SpreadsheetApp.openById('SPREADSHEET_ID').getSheetByName('SHEET_NAME');
let responses = form.getResponses();
let emails = sheet.getRange('EMAILS_RANGE').getValues();
let ids = sheet.getRange('IDS_RANGE').getValues();
let sent = sheet.getRange('SENT_RANGE').getValues();
for (let i = responses.length; i > 0; i--) {
let email = responses[i].getRespondentEmail();
for (let j = 0; j < emails.length; j++) {
if (emails[j][0] == email && sent[j][0] == false) {
// edit file permissions
// send email
// mark the associated row on the Sent col with TRUE
}
}
}
}
Trigger
As for making the function above run every time there's a new form submission, you will have to create an onFormSubmit trigger which will trigger the execution of the function.
Reference
Apps Script;
Apps Script Spreadsheet Service;
Apps Script Forms Service;
Apps Script Installable Triggers.
Using Google Sheets, I am trying to retrieve text passages from the Perseus Scaife Library, which has a working API.
When I query for the document node (=importxml("https://scaife-cts.perseus.org/api/cts?request=GetPassage&urn=urn:cts:greekLit:tlg0527.tlg001.opp-grc2:1.1","/")) I get all the data, including the URNs etc. However, any other xpath_query gives an error.
I know that Google Sheets can access the data, but I would like to be able to select only one node (//p).
You want to retrieve the text in passage. If my understanding is correct, how about this answer?
=importxml(A1, "//*[local-name()='passage']")
Result :
Note :
https://scaife-cts.perseus.org/api/cts?request=GetPassage&urn=urn:cts:greekLit:tlg0527.tlg001.opp-grc2:1.1 is converted by URL encode and put to "A1".
Converted URL is https://scaife-cts.perseus.org/api/cts?request=GetPassage&urn=urn%3acts%3agreekLit%3atlg0527%2etlg001%2eopp%2dgrc2%3a1%2e1.
Reference :
local-name
If this was not what you want, I'm sorry.
I work for a Google Apps Reseller and I implement gadgets for our clients.
We have several gadgets that use Google Visualization API to query a spreadsheet (with a dynamic id) and display the result in a table.
We are using this piece of code:
function drawVisualization() {
var oldSpreadsheetUrl = "https://docs.google.com/spreadsheet/tq?key=<ss_id>";
var newSpreadsheetUrl = "https://docs.google.com/spreadsheets/d/<ss_id>/gviz/tq";
var query = new google.visualization.Query(oldSpreadsheetUrl);
query.setQuery("SELECT A,B");
query.send(handleQueryResponse);
}
function handleQueryResponse(response) {
if(response.isError()) {
console.log(response.getMessage());
} else {
console.log("NUMBER OF COLUMNS: " + response.getDataTable().getNumberOfColumns());
}
}
And testing with this sample spreadsheet (New Google Spreadsheet)
SPREADSHEET ID: 1qMvY3y4MkwoK1UvCA9A8dW4BbmhH1iMwWqGLEM5vjJk
As long as we query an old spreadsheet everything works ok, but if we query a new spreadsheet (with the standard url - oldSpreadsheetUrl) the request fails with a timeout exception and we get a 404 exception.
This is the message we get in the Browser Console:
http://i.imgur.com/h4DjyQ6.png
We were able to make the query work anyway, by using this new url in the datasource:
"https://docs.google.com/spreadsheets/d/<spreadsheet_id>/gviz/tq"
But with this notation the setQuery directive is ignored and all of the three columns are returned instead of the first two ("SELECT A,B").
We are using this in several gadgets and therefore want to know:
How can we overcome the setQuery bug?
Is there a programmatic way to know if a spreadsheet is a new version or an old version?
Thank you in advance.
Regards,
Riccardo
I ran into this same problem a few weeks ago when we did some testing with the new Google Sheets. Basically, the old Google Sheets expect a URL with the following format:
https://spreadsheets.google.com/tq?key={key}
While the new Google Sheets expects a URL with this format:
https://docs.google.com/spreadsheets/d/{key}/gviz/tq
You should be able to use the domain to distinguish between old and new Google Sheets. Failing that, you could always look for /gviz/ to tell if the new Google Sheets are being used. You can find additional information in this bug report.
In regards to your other point about setQuery, I've just noticed this morning while debugging my own issue that column IDs are being returned by the Visualization API as "Col 0", "Col 1" etc. instead of "A", "B" etc. So that's likely why your ("SELECT A,B") query isn't working. I've also just logged a bug for this issue.
Hope that helps.
I have been trying to export data to a google bigquery dataset from datasift, but except for 4 empty rows, no other relevant data has been pushed.
I followed instruction from this link: http://dev.datasift.com/docs/push/connectors/bigquery. Not sure if it's the csdl code that I used the cause.
For example I configured a stream using:
wikipedia.title contains "Audi".
The live preview has no output. Also, the only data sources that I've set as active are Interaction and Wikipedia.
Please let me know what may be the reason for this. At the end of every stream recording I don't see any changes, expect the creation of the table mentioned in the destination with 4 empty rows(some row have null values, and interaction.type is ds_verify).
Thank you!
Ok, this one has become a little tricky for me and I really need some assistance to work through it.
Problem
I have a GSpreadsheet which has a list of data, in this case Twitter usernames. Using the API of a service provider (in this case the Klout API), I would like to retrieve information about that user to populate a cell within a spreadsheet.
Based on what I can work out so far, I would need to write a custom function to do this but I have no idea where to start, how I might construct it, or if there are any examples of doing this.
Scenario
The Klout API can return either an XML or JSON response (see http://developer.klout.com/docs/read/api/API), based on the string passed. For example, the URL:
http://api.klout.com/1/users/show.xml?key=SECRET&users=thewinchesterau
would return the following XML response:
<users>
<user>
<twitter_id>17439480</twitter_id>
<twitter_screen_name>thewinchesterau</twitter_screen_name>
<score>
<kscore>56.63</kscore>
<slope>0</slope>
<description>creates content that is spread throughout their network and drives discussions.</description>
<kclass_id>10</kclass_id>
<kclass>Socializer</kclass>
<kclass_description>You are the hub of social scenes and people count on you to find out what's happening. You are quick to connect people and readily share your social savvy. Your followers appreciate your network and generosity.</kclass_description>
<kscore_description>thewinchesterau has a low level ofinfluence.</kscore_description>
<network_score>58.06</network_score>
<amplification_score>29.16</amplification_score>
<true_reach>90</true_reach>
<delta_1day>0.3</delta_1day>
<delta_5day>0.5</delta_5day>
</score>
</user>
</users>
Based on this response, I would like to be able to populate different cells with the values returned within the XML (or JSON if easier) packet.
So, for example, I would have a spreadsheet like the following which would have custom functions to go out and retrieve the value of the relevant XML element response to populate the cell:
Cell A B C D E
1 Username kscore Network score Amplification score True reach
2 thewinchester =kscore(A2) =nscore(A2) =ascore(A2) =tscore(A2)
Questions
Are there any gSpreadsheet examples you know of that use an API to pull data in from an external source?
How would one write a custom function to fetch the result from the API and populate a cell with a result of a specific element?
Any information, examples or helpers you have are greatly appreciated.
You want the importXML function, documented here. The formula you want will look something like this:
=importXML("http://api.klout.com/1/users/show.xml?key=SECRET&users=" + A1, "//users/user/score/kscore")
You could write a custom script with Google AppScript, but there's a simple solution to this similar to what Nick Johnson posted. I've tested this against the score function, but it could be easily adapted to the show endpoint with different XPath.
=importXML("http://api.klout.com/1/klout.xml?users="&A1&"&key=YOUR_API_KEY", "//users/user/kscore")
This presumes your Twitter IDs are in the A column.
Note, Google Docs limits the number of such importXML functions to 50 per spreadsheet. You could concatenate groups of 5 userids for each importXML call, effectively putting your limit to 250 a sheet.
This could also be adapted to a similar call in Excel that doesn't have that limit. Keep in mind the Klout ToS, though, using proper attribution and rate limits.