As I know that, instead of get() I can use onSapshot() for my queries and listen to the changes(additions, deletions, modifications) for that query. But when there are changes, onSnaphot() returns multiple full documents whether modified, deleted or added (a new snapshot). What I want to do is, to check if new documents have been added matching my query and show a notification to the user that there are new records available, and only when a button is clicked, they should be able to fetch and see the new records. I don't want to fetch from the firestore whole sets of documents when there are changes. I just want to know about those changes and fetch on demand. And fetch only the added ones.
How can I do that? Any ideas?
By the way, I am using react-native for my app.
P.S. - I know, I can run the query periodically and check if the id of the first item in the new query matches the first item id in the previous query, and so I can detect the added records and show my notification/button. But I am looking for a more elegant solution. I think the notification should be triggered from the backend instead of polling the backend periodically.
P.S. 2 - Using cloud functions would not seem to be a logical option since these queries will be different for each user of my app. Which would require running thousands of functions (hopefully more) on the firestore. Or would it?
Thanks!
There is no way in the Firestore API to get notified about changes to a query without actually retrieving the changed documents. But you can of course show just a notification to the user when your onSnapshot callback gets called, and then only show the actual data from those documents when the user chooses to refresh the UI.
On a second note, when you use a snapshot listener the Firestore client will only retrieve the modified documents on additional callbacks.
Say you attach a listener for a query that matches 10 documents. On the first onSnapshot callback, you will get 10 documents and will be charged for 10 document reads. Now say that one of the documents changes. When your onSnapshot callback gets invoked for this, you will see 10 documents again, but will be charged only 1 read - for the document that was changed.
If you only want to process the changes, have a look at the documentation on viewing changes between snapshots, which contains a good example of how to do this with the docChanges() method.
Related
An XPage is used to display the number of points a person has collected, and the number of points remaining (see below).
I have a repeat control which gets a collection of documents meeting a specific criteria. The last column in the control contains 5 digit/form/comboboxes, which are displayed or hidden, according to the number of fields on each document that contain data.
The layout contains gift cards worth a certain amount of points, and the person can select how many of each gift card they want. eg.
Company Available in Values of Points Required Quantity Requested
The Quantity Requested column contains the digit/form/comboboxes. As the person selects values in the checkbox, I want the number of points remaining to be recalculated.
The onChange event of the digit/form/comboboxes calls a function in an Output Script which calls an RPC, which in turn calls an SSJS function. The SSJS function cycles through the documents displayed in the repeat control, gathering the points required information. I then wanted it to also grab the Quantity Requested. I understand from a previous posting that because of the way the digit/form/combox is rendered, I can only get the value using CSJS with dijit.byId and perhaps putting the value in a hidden field and retrieving it from there.
I can't seem to wrap my head around how I will do this when the repeat control will make it possible for there to be many combobox1 and combobox2, etc.
The XPage is not bound to a form, because all the items are just calculated on the fly and then discarded.
What is the best way to do this?
The JSON RPC service can't interact with any changes made in the browser, see https://www.intec.co.uk/json-rpc-service-component-tree-manipulation-openlog/. This could be the cause of your problems.
You may be able to get around it by triggering a partial refresh (POST) before calling the JSON RPC. In theory that might work, because the component tree (server-side map of the XPage) would get updated by the partialRefreshPost and the updates picked up by the JSON RPC. It's possible though that the Restore View picks up a version of the XPage other than the one for the browser, I don't know. I've never investigated that.
It's been a while since I've worked with server java script, I have been doing it the managed bean way with ActionListeners. If you have the data in the UI, then can you avoid server side processing and do it client side?
You can also use the DOM XSP Object like XSP.setSubmittedValue to have a key value pair sent with your post request to the server side, you can only have one... it can be JSON or any other value you set it to from the client side javascript.
I figured out how to do this. If anyone wants the code, let me know and I'll provide it.
I am developing an app with the Active Collab API using the What's New endpoint.
I am retrieving this regularly, so I have the latest information. I was wondering if there was a way to specify the activity by date to get the items since then?
For example, instead of getting 50 records and processing the 50 every time, if I could pass a from parameter (with a timestamp) to only collate activity since that time then that would help with both my processing and the size of the request (and knowing how many things have happened since)
Is this possible?
What's New API end point only has a daily filter:
/whats-new/daily/YYYY-MM-DD
Both global and daily What's new API end-points are paginated, so you can loop through responses by providing (and incrementing) page GET argument:
/whats-new?page=2
until you reach records that are older than the timestamp that you are looking for (or get an empty result). At that point, you just break and you have all the updates that were looking for.
GitHub API provides the feature of activity events for users, orgs and repos. The APIs support pagination upto 10 pages for a total of 300 events with 30 events per page. Rate Limiting is achieved using ETAG headers. I am trying to poll this API to get the latest activity. However this scheme is very in-efficient due to the design supported by Github as mentioned. Lets say I make a request on page-1 by
https://api.github.com/users/me/events/orgs/my-org?page=1
and i will get an ETAG entry for this page. Now I move to the next page-2 and do
https://api.github.com/users/me/events/orgs/my-org?page=2
and will get the ETAG for this 2nd page. Similarly I can pull events from all 10 supported pages.
Now lets say that some activity was performed on my orgs Github account. Lets assume that only 1 new event occured. In this case when I poll the API for page-1 with the ETAG it will return the changed page with the new event included in it. Similarly polling on page-2 with its previous ETAG will also send the changed page. This change in page-2 is however the event that was previously the last event of page-1 and has now moved to the top on page-2. This "shift-to-next" will happen for all the pages. There is NO way to find out the number of NEW events that took place.The only solution is to keep polling on page-1 to get the latest events. However this approach has a serious flaw explained below:
The situation gets worse when the number of new events between my poll rounds is greater than 30(max items on one page). In this case, events prior to the latest new 30 events will slip to page-2 directly. If I only poll on page-1 i will loose these events that slipped to page-2. The only solution that is coming to my mind is to keep a cache of the entire events and then sweep on all pages. This is however a very in-efficient and un-desirable way to do it and kills the purpose of on events notification API.
I hope some github-dev can answer this
Since each event has an ID and events are ordered in the response, you only need to remember the ID of the first event in the previous response (not all of the events).
So, the way I would do it is:
Initial fetch:
fetch all event pages (pages from 1 to 10)
store the ETAG of the first page
store the ID of the first event in the first page
Subsequent fetches:
conditionally fetch first page of events with the stored ETAG
if a 304 Not modified response is received, then there are no new events so terminate
if a 200 OK response is received, then we have new events. Fetch pages from 1 to 10 sequentially until the first page that contains the event with the ID equal to the stored ID. All newly fetched events up until that event are new events and should be processed. So, the number of new events is discovered incrementally as the result of fetching all events up until the event you have seen before. And you are fetching only pages that you have to fetch, not more than that.
store the ETAG of the first page
store the ID of the first event in the first page
wait for some time and then go to step 1
I'm trying to aggregate some information about the kanban states of my user stories. If query a PifTeam item, I get a summarized collection of UserStories associated with it.
Example query:
https://rally1.rallydev.com/slm/webservice/1.40/portfolioitem/pifteam/99999999999.js
However I then have to run a loop on the UserStories collection, individually querying each one to get at the information I need. This potentially results in a lot of web service calls.
Is there a way to return the full hierarchical requirement information in the original pifteam query so that there is only one webservice call which returns all sub-objects? I read the webservice api and was trying to play with the fetch parameter but had no success.
This functionality will be disabled in WSAPI 2.0 but will continue to be available in the 1.x versions. That said, you should be able to use a fetch the fields on story that you need like this:
/pifteam/9999.js?fetch=UserStories,FormattedID,Name,PlanEstimate,KanbanState
Fetch will hydrate the fields specified on sub objects even if the root object type doesn't have those fields. So by fetching UserStories the returned collection will populated with stories, each having the FormattedID, Name, PlanEstimate and KanbanState fields included.
There is no way to do it from Rally's standard Web Services API (WSAPI) but you can from the new Lookback API (LBAPI). The query would look something like this:
https://rally1.rallydev.com/analytics/v2.0/service/rally/workspace/<ObjectID_for_Workspace>/artifact/snapshot/query.js?find={__At:"current",_TypeHierarchy:"HierarchicalRequirement",Children:null,_ItemHierarchy:<ObjectID_for_PortfolioItem>}&fields=["Name"]
Fill in the ObjectIDs for your Workspace and PortfolioItem. The _ItemHierarchy field will cross work item type boundaries and goes all the way from PortfolioItems down through the Story hierarchy down to Defects and even Tasks, so I added _TypeHierarchy:"HierarchicalRequirement" to limit it to Stories. I have specified Children:null which means you'll only get back leaf Stories. The __At:"current" clause get's the current tree and values. Remember, it's the "Lookback" API, so you can retrieve the state of the object at any moment in history. __At:"current" says to get the current values and tree.
Note, the LBAPI is delayed from current values in the system by anywhere from seconds to minutes. Typically it's about 30 seconds behind. You can see how far behind it is by checking the ETLDate field in the response.
Details about the LBAPI can be found here. Note, that the LBAPI is available in preview now for almost all Rally customers. There are still a number of customers where it is not yet turned on. The best way to tell if it's working for your subscription is to try the query.
for writing an offline client to the Google Reader service I would like to know how to best sync with the service.
There doesn't seem to be official documentation yet and the best source I found so far is this: http://code.google.com/p/pyrfeed/wiki/GoogleReaderAPI
Now consider this: With the information from above I can download all unread items, I can specify how many items to download and using the atom-id I can detect duplicate entries that I already downloaded.
What's missing for me is a way to specify that I just want the updates since my last sync.
I can say give me the 10 (parameter n=10) latest (parameter r=d) entries. If I specify the parameter r=o (date ascending) then I can also specify parameter ot=[last time of sync], but only then and the ascending order doesn't make any sense when I just want to read some items versus all items.
Any idea how to solve that without downloading all items again and just rejecting duplicates? Not a very economic way of polling.
Someone proposed that I can specify that I only want the unread entries. But to make that solution work in the way that Google Reader will not offer this entries again, I would need to mark them as read. In turn that would mean that I need to keep my own read/unread state on the client and that the entries are already marked as read when the user logs on to the online version of Google Reader. That doesn't work for me.
Cheers,
Mariano
To get the latest entries, use the standard from-newest-date-descending download, which will start from the latest entries. You will receive a "continuation" token in the XML result, looking something like this:
<gr:continuation>CArhxxjRmNsC</gr:continuation>`
Scan through the results, pulling out anything new to you. You should find that either all results are new, or everything up to a point is new, and all after that are already known to you.
In the latter case, you're done, but in the former you need to find the new stuff older than what you've already retrieved. Do this by using the continuation to get the results starting from just after the last result in the set you just retrieved by passing it in the GET request as the c parameter, e.g.:
http://www.google.com/reader/atom/user/-/state/com.google/reading-list?c=CArhxxjRmNsC
Continue this way until you have everything.
The n parameter, which is a count of the number of items to retrieve, works well with this, and you can change it as you go. If the frequency of checking is user-set, and thus could be very frequent or very rare, you can use an adaptive algorithm to reduce network traffic and your processing load. Initially request a small number of the latest entries, say five (add n=5 to the URL of your GET request). If all are new, in the next request,
where you use the continuation, ask for a larger number, say, 20. If those are still all new, either the feed has a lot of updates or it's been a while, so continue on in groups of 100 or whatever.
However, and correct me if I'm wrong here, you also want to know, after you've downloaded an item, whether its state changes from "unread" to "read" due to the person reading it using the Google Reader interface.
One approach to this would be:
Update the status on google of any items that have been read locally.
Check and save the unread count for the feed. (You want to do this before the next step, so that you guarantee that new items have not arrived between your download of the newest items and the time you check the read count.)
Download the latest items.
Calculate your read count, and compare that to google's. If the feed has a higher read count than you calculated, you know that something's been read on google.
If something has been read on google, start downloading read items and comparing them with your database of unread items. You'll find some items that google says are read that your database claims are unread; update these. Continue doing so until you've found a number of these items equal to the difference between your read count and google's, or until the downloads get unreasonable.
If you didn't find all of the read items, c'est la vie; record the number remaining as an "unfound unread" total which you also need to include in your next calculation of the local number you think are unread.
If the user subscribes to a lot of different blogs, it's also likely he labels them extensively, so you can do this whole thing on a per-label basis rather than for the entire feed, which should help keep the amount of data down, since you won't need to do any transfers for labels where the user didn't read anything new on google reader.
This whole scheme can be applied to other statuses, such as starred or unstarred, as well.
Now, as you say, this
...would mean that I need to keep my own read/unread state on the client and that the entries are already marked as read when the user logs on to the online version of Google Reader. That doesn't work for me.
True enough. Neither keeping a local read/unread state (since you're keeping a database of all of the items anyway) nor marking items read in google (which the API supports) seems very difficult, so why doesn't this work for you?
There is one further hitch, however: the user may mark something read as unread on google. This throws a bit of a wrench into the system. My suggestion there, if you really want to try to take care of this, is to assume that the user in general will be touching only more recent stuff, and download the latest couple hundred or so items every time, checking the status on all of them. (This isn't all that bad; downloading 100 items took me anywhere from 0.3s for 300KB, to 2.5s for 2.5MB, albeit on a very fast broadband connection.)
Again, if the user has a large number of subscriptions, he's also probably got a reasonably large number of labels, so doing this on a per-label basis will speed things up. I'd suggest, actually, that not only do you check on a per-label basis, but you also spread out the checks, checking a single label each minute rather than everything once every twenty minutes. You can also do this "big check" for status changes on older items less often than you do a "new stuff" check, perhaps once every few hours, if you want to keep bandwidth down.
This is a bit of bandwidth hog, mainly because you need to download the full article from Google merely to check the status. Unfortunately, I can't see any way around that in the API docs that we have available to us. My only real advice is to minimize the checking of status on non-new items.
The Google API hasn't yet been released, at which point this answer may change.
Currently, you would have to call the API and dis-regard items already downloaded, which as you said isn't terribly efficient as you will be re-downloading items every time, even if you already have them.