Is there a way to query using the LookbackAPI to find all the stories (HierarchicalRequirement) that had a change to PlanEstimate from a certain date (say a week ago) to today?
Or is the best way to do that to find all stories on a given date in the past and return the PlanEstimate and UnformatedID fields, and then compare them to the current PlanEstimate for each story as of today?
Each LBAPI Snapshot will have a _PreviousValues collection. For each LBAPI Snapshot, if a field has changed from the prior snapshot, then the _PreviousValues collection will be hydrated for that value. In other words, if the PlanEstimate has not changed between snapshots, then there will be no PlanEstimate entry in the _PreviousValues collection for that snapshot. Thus, for PlanEstimate, the following query:
find: {
"_TypeHierarchy" : "HierarchicalRequirement",
"_PreviousValues.PlanEstimate" : {$exists: true},
"_ValidFrom": {
$gte: "2013-01-01T00:00:00.000Z",
$lt: "2013-05-01T00:00:00.000Z"
}
}
Should provide you with any HierarchicalRequirement snapshots corresponding to a change in PlanEstimate from 2013-05-01 to 2013-05-10.
Related
In our raven based application we are starting to experience major performance issues when the master document starts increasing in size, as it holds a lot of collections that keep growing. As such I am now planning a major data redesign that is likely to take months and I want to be sure I'm on the right track before I do so.
The current design looks like this:
Community
{
id:,
name,
//other properties
members[
{
id:,
name:,
date of birth:
//etc
},
{
//another member, this list could potentially grow to hundreds of thousands
}
],
league
[
{
id,
name,
seasons[
{...},
{
id:,
divisions
[
{
id:,
name:
matches[
{
id:,
//match details
},
{
//another match. there could be hundreds here in a big league
},
{}
As we started hitting performance issues, we started using transformers to only load what is needed, but that didn't solve the problem fully as some of our leagues are a couple mb's just on their own. The other issue is we always need to be doing member checks to check for admin/membership rights so the members list is always needed.
I understand I could omit the member list completely using a transformer and use an index for membership checks, but the problem remains about what to do, when a member is added , that list will need to be loaded and with an upcoming project there is a potential for it to grow to half a million people or more.
So my plan is separate each entity into it's own document, so in the case of leagues, I will have a league document, and a match document, with matches containing {leagueId, season number, division number, other match details}.
Each member will have their own document with a list of community document Id's they're a member of.
I'm just a bit worried, that using this design, is missing the whole point of a document db and we may as well have used sql, or do you think I'm on the right track with approach?
According to this page in the docs, only one snapshot can be active for a given object. However, I seem to have a Defect with 2 active snapshots. All snapshots are shown in the screenshot below:
As you can, see I have connected the snapshots with arrows and they do not all link together. Is this a bug with Rally or is it in fact possible to have 2 defects with _ValidTo dates in the year 9999?
My query is taken from the example in the docs:
URI: https://rally1.rallydev.com/analytics/v2.0/service/rally/workspace/12345/artifact/snapshot/query.js
POST data:
{
"find": {
"ObjectID": my funky object
},
"fields": ["State", "_ValidFrom", "_ValidTo", "ObjectID", "FormattedID"],
"hydrate": ["State"],
"compress": true
}
The object should not have two current snapshots with _ValidTo set to 9999-01-01. Please contact CA Agile Central (Rally) support, and they will raise the issue with the LookbackAPI team, which I believe has a way of fixing the data on their end.
I am trying to use the lookback api to get stories that have the Ready flag set to true. I appear to be able to output the ready flag but not filter on it.
I do not appear to be able to filter or output the recycled flag.
Curious if these fields are not available to the lookback api as filters.
Nick is right about the Recycled artifacts. I'm wondering though if you are quoting the Ready values and treating them like a String. This field is actually a Boolean and can be queried like this:
https://rally1.rallydev.com/analytics/v2.0/service/rally/workspace/12352608129/artifact/snapshot/query.js?find={"Ready":true}&fields=true
This doesn't work (since it's matching on the string "true" instead of the Boolean true).
https://rally1.rallydev.com/analytics/v2.0/service/rally/workspace/12352608129/artifact/snapshot/query.js?find={"Ready":"true"}&fields=true
It's also listed as #1 in the Tips and Tricks section of the documentation:
https://rally1.rallydev.com/analytics/doc/Analytics2.0LookbackAPIUserManual.html#h.ce80nb6raonb
I believe you're correct. In LBAPI documentation examples that illustrate the temporal data model show a gap when artifacts were moved to a recycle bin and later restored, or display no data after artifacts were moved to recycle bin and never restored.
Per this SO post, Lookback API doesn't provide anything along the lines of tracking entries in the Recycle Bin.
As far as Ready flag, a query like this will return the field value:
https://rally1.rallydev.com/analytics/v2.0/service/rally/workspace/12352608129/artifact/snapshot/query.js?find={"ObjectID":{$in:[16483705391,14167827399]}}&fields=["Name","Ready"]
e.g.
{
Name: "my story",
Ready: false
},
{
Name: "my story",
Ready: true
},
and this query will return results if value true is treated as Boolean, not string, as SRMelody pointed out:
https://rally1.rallydev.com/analytics/v2.0/service/rally/workspace/12352608129/artifact/snapshot/query.js?find={"Ready":true}&fields=true
I've been tasked with plotting some defect charts using Rally historical data. Right now I'm using a simple REST client to pull data at certain points in time and plot the count on a spreadsheet. What I'm doing right now is:
{
find : {
"_ProjectHierarchy": <projectId>,
"_TypeHierarchy": -51006,
"FoundInBuild" : {$regex: "3\\.3\\."},
"State" : {$in : ["Submitted","Open"] },
$or: [
{"Severity" : { $in : ["Catastrophic","Severe"] }},
{"Priority" : "showstopper"}
],
"__At" : "<date>"
},
pagesize : 1,false
}
I just run this once for every date I need the data for. That's a lot of queries! What I'm looking for is a way to run a single query using _ValidFrom and _ValidTo to enclose a time range, then pass it on to a SnapshotStore, then plot that on a Chart? I'm certain there's a way to do it, but I can't figure it out from the docs. Any help much appreciated.
Unfortunately, the example space for AppSDK2 and Lookback API is presently a bit thin. There are some cool apps out there, for instance, you may wish to check out David Thomas' cool Hackathon app:
Defect Re-work Trend
As a starting point. It queries LBAPI for Defects and stores the resulting data in a SnapshotStore. The App itself measures Defect "thrash" or the trending around how many times a Defect is re-opened during a particular development cycle.
In reviewing Hackathon apps, just be aware that certain methods and syntax for the SnapshotStore may change slightly in a future release of AppSDK2.
I'm considering using MongoDB or CouchDB on a project that needs to maintain historical records. But I'm not sure how difficult it will be to store historical data in these databases.
For example, in his book "Developing Time-Oriented Database Applications in SQL," Richard Snodgrass points out tools for retrieving the state of data as of a particular instant, and he points out how to create schemas that allow for robust data manipulation (i.e. data manipulation that makes invalid data entry difficult).
Are there tools or libraries out there that make it easier to query, manipulate, or define temporal/historical structures for key-value stores?
edit:
Note that from what I hear, the 'version' data that CouchDB stores is erased during normal use, and since I would need to maintain historical data, I don't think that's a viable solution.
P.S. Here's a similar question that was never answered: key-value-store-for-time-series-data
There are a couple options if you wanted to store the data in MongoDB. You could just store each version as a separate document, as then you can query to get the object at a certain time, the object at all times, objects over ranges of time, etc. Each document would look something like:
{
object : whatever,
date : new Date()
}
You could store all the versions of a document in the document itself, as mikeal suggested, using updates to push the object itself into a history array. In Mongo, this would look like:
db.foo.update({object: obj._id}, {$push : {history : {date : new Date(), object : obj}}})
// make changes to obj
...
db.foo.update({object: obj._id}, {$push : {history : {date : new Date(), object : obj}}})
A cooler (I think) and more space-efficient way, although less time-efficient, might be to store a history in the object itself about what changed in the object at each time. Then you could replay the history to build the object at a certain time. For instance, you could have:
{
object : startingObj,
history : [
{ date : d1, addField : { x : 3 } },
{ date : d2, changeField : { z : 7 } },
{ date : d3, removeField : "x" },
...
]
}
Then, if you wanted to see what the object looked like between time d2 and d3, you could take the startingObj, add the field x with the value 3, set the field z to the value of 7, and that would be the object at that time.
Whenever the object changed, you could atomically push actions to the history array:
db.foo.update({object : startingObj}, {$push : {history : {date : new Date(), removeField : "x"}}})
Yes, in CouchDB the revisions of a document are there for replication and are usually lost during compaction. I think UbuntuOne did something to keep them around longer but I'm not sure exactly what they did.
I have a document that I need the historical data on and this is what I do.
In CouchDB I have an _update function. The document has a "history" attribute which is an array. Each time I call the _update function to update the document I append to the history array the current document (minus the history attribute) then I update the document with the changes in the request body. This way I have the entire revision history of the document.
This is a little heavy for large documents, there are some javascript diff tools I was investigating and thinking about only storing the diff between the documents but haven't done it yet.
http://wiki.apache.org/couchdb/How_to_intercept_document_updates_and_perform_additional_server-side_processing
Hope that helps.
I can't speak for mongodb but for couchdb it all really hinges on how you write your views.
I don't know the specifics of what you need but if you have a unique id for a document throughout its lifetime and store a timestamp in that document then you have everything you need for robust querying of that document.
For instance:
document structure:
{ "docid" : "doc1", "ts" : <unix epoch> ...<set of key value pairs> }
map function:
function (doc) {
if (doc.docid && doc.ts)
emit([doc.docid, doc.ts], doc);
}
}
The view will now output each doc and its revisions in historical order like so:
["doc1", 1234567], ["doc1", 1234568], ["doc2", 1234567], ["doc2", 1234568]
You can use view collation and start_key or end_key to restrict the returned documents.
start_key=["doc1", 1] end_key=["doc1", 9999999999999]
will return all historical copies of doc1
start_key=["doc2", 1234567] end_key=["doc2", 123456715]
will return all historical copies of doc2 between 1234567 and 123456715 unix epoch times.
see ViewCollation for more details