I . sorry for posting this second or third time but couldn't find any solutions from the community. let me know if some who can help me out. Any power query expert here with Rest API's who can help me with List. Generate function query for pagination?
Issue: Pagination of API URL that allows 50 rows of data per Agent.
URL: https://vcc-na4.8x8.com/api/stats/agents/{Agent-ID}/Activities?n=1
Background: Created a function below which allows me to extract all Agent ID's(But only 50 rows per ID).
(id as text, n as number) as table=>
let
Source = Xml.Tables(Web.Contents("https://vcc-na8.8x8.com",
[RelativePath="/api/stats/agents/" &(id)&"/activities?n="&number.totext(n) ) ),
in
Source
Than a created a query below to extract all the rows as well watching curbal's video,
Video Link: https://www.youtube.com/watch?v=05yhwnuCjRw
Query:
let
Source = List.Generate(()=>
[result = try all_rows (1) otherwise try null, n=1],
each [result]<> null,
each [result = try all_rows ([n]+50) otherwise null, n=[n]+50],
each [result]
)
in
Source
but it shows this error,
enter image description here
Related
I'm building a report in Power BI and could setup a Power Query custom column using Text.Contains to solve this problem but the M Code would be very long and I'd rather perform this upstream in the SQL query. I have very little SQL experience.
I'm working with website data from Adobe Analytics. We have our website URLS and web pages grouped into categorical segments based on the product/service the URL/webpage corresponds to. A segment is defined by a list of URL paths and/or web page names, sometimes 1 path/page, sometimes over 30.
My result needs to be the following table:
Page URL Path
Page Name
Page Category
varchar(255)
varchar(255)
varchar(255)
Page URL Path examples:
/careers/starting-your-career/scholarships.html
/services/technology/ecommerce.html
Corresponding Page Name Examples:
Career & Scholarships | Company Name
Digital Transformation | E-Commerce | Company Name
There are a total of 76 page categories/segments to define. This screenshot shows an example of some categories and their definition.
Can anyone help me get started in writing this query?
I tried using CONTAINS but I believe this only works within a WHERE statement and I don't think it can be scaled to the needed extent:
SELECT
post_evar3 as 'Page URL Path',
post_evar4 as 'Page Name',
CASE
WHEN post_evar3 CONTAINS ('/services/assurance' or 'services/audit' or 'insights/financial-reporting')
AND (post_evar3 CONTAINS 'asc-842' OR post_evar4 CONTAINS 'asc 842')
THEN 'Audit Services'
WHEN post_evar3 CONTAINS '/services/strategy-and-management-consulting'
THEN 'Business Stratgegy Operations'
ELSE 'Other'
END AS 'Page Category'
FROM
Marketing.WebAnalytics.WebData
WHERE
exclude_hit = 0
AND hit_source = 1
I've read about Full-Text Search and Index solutions that are over my head in developing and I don't know that this method can be used within the Power BI SQL query environment. I've wondered if I need to declare the definition values into their own table, then join with the WebData table, though defining using both Page URL Path AND Page Name for the same category throws me for a loop.
The M code for this kind of matching is not large, though execution time can can vary
let BufferedTable2=Table.Buffer(Table2),
Source = Table.AddColumn(Table1,"Match",(i)=>try Table.SelectRows( BufferedTable2, each Text.Contains(i[Column1],[Match1], Comparer.OrdinalIgnoreCase) and Text.Contains(i[Column2],[Match2], Comparer.OrdinalIgnoreCase) ) [Return]{0} otherwise null, type text)
in Source
I have succeeded using Thunkable to archive old data in a Fusion Table. I would like this to be done in the background of the app using Google Apps Script.
The Thunkable Blocks with SQL is as follows:
Query 1:
SELECT ROWID FROM TableID WHERE Duration<= Clock.Now
SET GLOBAL RESULTS to List from CSV Table text (Result from Query1)
For each number from 2 to length of list by 1 DO Query 2
Query 2:
UPDATE TableID SET Availability='uNAVAILABLE' WHERE ROWID='list item 2 from result from Query 1'
Remove list item 2
Query 3:
DELETE FROM TableID WHERE Availability='Unavailable'
How can I convert this to Google Apps Script and link it to a Fusion Table? Thank you.
Per documentation,
A quick way to try out the API is to type the command or query directly into your browser's toolbar. You can adjust the URL as you change your query or data needs and you'll get immediate feedback. You can only do this with tables that are exportable and either public or unlisted, and you need to include your API key.
Here's a sample that runs a query to select all rows in a given table,
https://www.googleapis.com/fusiontables/v2/query?sql=SELECT * FROM 1KxVV0wQXhxhMScSDuqr-0Ebf0YEt4m4xzVplKd4&key={your API key}
So, to implement this using Google Apps Script, try using Class UrlFetchApp.
Fetch resources and communicate with other hosts over the Internet. This service allows scripts to communicate with other applications or access other resources on the web by fetching URLs. A script can use the URL Fetch service to issue HTTP and HTTPS requests and receive responses.
You may want to check this sample Google Apps script and fusion table query in this GitHub post for additional insights.
function readFacName(fac, city){
// public fusion table
// https://www.google.com/fusiontables/DataSource?docid=1tL67aacGcCyMfAg9PUo_-gp4qm74GDtFiCMtFg
var select = "select FACNAME from 1tL67aacGcCyMfAg9PUo_-gp4qm74GDtFiCMtFg ";
var where = "where FAC_ZIP5 = '" + fac + "' AND FAC_CITY = '" + city +"'";
var query = encodeURIComponent(select + where);
var url = "http://www.google.com/fusiontables/api/query?sql=" + query;
var response = UrlFetchApp.fetch(url, {method: "get"});
return response.getContentText();
}
function fTable() {
Logger.log(readFacName("94609","OAKLAND"));
}
I have an Excel file in which are given more then 2000 organisms, where each one of them has a Bioproject ID associated (like PRJNA12997). The idea is to use these IDs to get the sequence for a later multiple alignment with other five sequences that I have in a text file.
Can anyone help me understand how I can do this using biopython? At least the part with the bioproject ID.
You can first get the info using Bio.Entrez:
from Bio import Entrez
Entrez.email = "Your.Name.Here#example.org"
# This call to efetch fails sometimes with a 400 error.
handle = Entrez.efetch(db="bioproject", id="PRJNA12997")
I've been trying, and Entrez.read(handle) doesn't seems to work. But if you do record_xml = handle.read() you'll get the XML entry for this record. In this XML you can get the ID for the organism, in this case 12997.
handle = Entrez.esearch(db="nuccore", term="12997[BioProject]")
search_results = Entrez.read(handle)
Now you can efecth from your search results. At this point you should use Biopython to parse whatever you will get in the efetch step, playing with the rettype http://www.ncbi.nlm.nih.gov/books/NBK25499/table/chapter4.T._valid_values_of__retmode_and/
for result in search_results["IdList"]:
entry = Entrez.efetch(db="nuccore", id=result, rettype="fasta")
this_seq_in_fasta = entry.read()
I've built an app using twitter4j which pulls in a bunch of tweets when I enter a keyword, takes the geolocation out of the tweet (or falls back to profile location) then maps them using ammaps. The problem is I'm only getting a small portion of tweets, is there some kind of limit here? I've got a DB going collecting the tweet data so soon enough it will have a decent amount, but I'm curious as to why I'm only getting tweets within the last 12 hours or so?
For example if I search by my username I only get one tweet, that I sent today.
Thanks for any info!
EDIT: I understand twitter doesn't allow public access to the firehose.. more of why am I limited to only finding tweets of recent?
You need to keep redoing the query, resetting the maxId every time, until you get nothing back. You can also use setSince and setUntil.
An example:
Query query = new Query();
query.setCount(DEFAULT_QUERY_COUNT);
query.setLang("en");
// set the bounding dates
query.setSince(sdf.format(startDate));
query.setUntil(sdf.format(endDate));
QueryResult result = searchWithRetry(twitter, query); // searchWithRetry is my function that deals with rate limits
while (result.getTweets().size() != 0) {
List<Status> tweets = result.getTweets();
System.out.print("# Tweets:\t" + tweets.size());
Long minId = Long.MAX_VALUE;
for (Status tweet : tweets) {
// do stuff here
if (tweet.getId() < minId)
minId = tweet.getId();
}
query.setMaxId(minId-1);
result = searchWithRetry(twitter, query);
}
Really it depend on which API system you are using. I mean Streaming or Search API. In the search API there is a parameter (result_type) that is an optional parameter. The values of this parameter might be followings:
* mixed: Include both popular and real time results in the response.
* recent: return only the most recent results in the response
* popular: return only the most popular results in the response.
The default one is the mixed one.
As far as I understand, you are using the recent one, that is why; you are getting the recent set of tweets. Another issue is getting low volume of tweets that have the geological information. Because there are very few users added the geological information to their profile, you are getting very few tweets.
I use jqGrid to display data which is retrieved using NHibernate. jqGrid does paging for me, I just tell NHibernate to get "count" rows starting from "n".
Also, I would like to highlight specific record. For example, in list of employees I'd like a specific employee (id) to be shown and pre-selected in table.
The problem is that this employee may be on non-current page. E.g. I display 20 rows from 0, but "highlighted" employee is #25 and is on second page.
It is possible to pass initial page to jqGrid, so, if I somehow use NHibernate to find what page the "highlighted" employee is on, it will just navigate to that page and then I'll use .setSelection(id) method of jqGrid.
So, the problem is narrowed down to this one: given specific search query like the one below, how do I tell NHibernate to calculate the page where the "highlighted" employee is?
A sample query (simplified):
var query = Session.CreateCriteria<T>();
foreach (var sr in request.SearchFields)
query = query.Add(Expression.Like(sr.Key, "%" + sr.Value + "%"));
query.SetFirstResult((request.Page - 1) * request.Rows)
query.SetMaxResults(request.Rows)
Here, I need to alter (calculate) request.Page so that it points to the page where request.SelectedId is.
Also, one interesting thing is, if sort order is not defined, will I get the same results when I run the search query twice? I'd say that SQL Server may optimize query because order is not defined... in which case I'll only get predictable result if I pull ALL query data once, and then will programmatically in C# slice the specified portion of query results - so that no second query occur. But it will be much slower, of course.
Or, is there another way?
Pretty sure you'd have to figure out the page with another query. This would surely require you to define the column to order by. You'll need to get the order by and restriction working together to count the rows before that particular id. Once you have the number of rows before your id, you can figure what page you need to select and perform the usual paging query.
OK, so currently I do this:
var iquery = GetPagedCriteria<T>(request, true)
.SetProjection(Projections.Property("Id"));
var ids = iquery.List<Guid>();
var index = ids.IndexOf(new Guid(request.SelectedId));
if (index >= 0)
request.Page = index / request.Rows + 1;
and in jqGrid setup options
url: "${Url.Href<MyController>(c => c.JsonIndex(null))}?_SelectedId=${Id}",
// remove _SelectedId from url once loaded because we only need to find its page once
gridComplete: function() {
$("#grid").setGridParam({url: "${Url.Href<MyController>(c => c.JsonIndex(null))}"});
},
loadComplete: function() {
$("#grid").setSelection("${Id}");
}
That is, in request I lookup for index of id and set page if found (jqGrid even understands to display the appropriate page number in the pager because I return the page number to in in json data). In grid setup, I setup url to include the lookup id first, but after grid is loaded I remove it from url so that prev/next buttons work. However I always try to highlight the selected id in the grid.
And of course I always use sorting or the method won't work.
One problem still exists is that I pull all ids from db which is a bit of performance hit. If someone can tell how to find index of the id in the filtered/sorted query I'd accept the answer (since that's the real problem); if no then I'll accept my own answer ;-)
UPDATE: hm, if I sort by id initially I'll be able to use the technique like "SELECT COUNT(*) ... WHERE id < selectedid". This will eliminate the "pull ids" problem... but I'd like to sort by name initially, anyway.
UPDATE: after implemented, I've found a neat side-effect of this technique... when sorting, the active/selected item is preserved ;-) This works if _SelectedId is reset only when page is changed, not when grid is loaded.
UPDATE: here's sources that include the above technique: http://sprokhorenko.blogspot.com/2010/01/jqgrid-mvc-new-version-sources.html