Looking for a data visualization framework [closed] - data-visualization

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 8 years ago.
Improve this question
(I know there are post about data visualization, but none of them completely answered my question.)
I am looking for a tool to generate charts from data. I want to tool that is be able to do some computation for me, not only show the data a nice way.
For example here is data from tests I run:
{
"destination" : "30be317c-e586-4448-8fed-1a9481710670",
"source" : "30be317c-e586-4448-8fed-1a9481710670",
"userStatus" : "UNDEF",
"testName" : "VolumeUp",
"testStatus" : "WORKED",
"testStart" : 1323378809108,
"testEnd" : 1323378809108
}, {
"destination" : "30be317c-e586-4448-8fed-1a9481710670",
"source" : "30be317c-e586-4448-8fed-1a9481710670",
"userStatus" : "FAILED",
"testName" : "VolumeDown",
"testStatus" : "FAILED",
"testStart" : 1323378814065,
"testEnd" : 1323378816077
}
Note this is currently in JSON, but I can transform it to XML or push it to a database if needed.
What I would like to have is a way to tell the tool:
Create a table with:
Column one: number of "testStatus" : "WORKED"
Column two: number of "testStatus" : "FAILED"
Column three: percentage between one and two.
Rows: "testName"
Show the percentage higher than 90% in green
The calculations remain very basic.
I did some research and found tools that do it (JasperReport, SpagoBI, RapidMiner) but all of them seem to be too much of a pain to generate easy reports like mine.
Is there a tool/framework that could help me doing that?
Or will I have to do the calculations by myself and use a tool like Google Visualization API to show the output?
Thanks

I am not sure if understand correctly.
I understand that you have some data in json format. You can convert this data to xml or load to a database. And you need to visualize this data with special calculations and/or conditional formatting.
The first tool you can use is Excel. You can use Excel's built-in functions to process and visualize your data. I believe that Excel will be enough for your requirements.
If you are going to design a cube with your data you have 2 options. First, you can make these calculations at cube level or you can use the features of the tool you use to visualize your data.
If you design a cube, you can also use Excel. But if you need a more advanced tool, my first suggestion will be TARGIT because of it's ease of use. You can also use QlikView or Tableau. For more advanced solutions, you can go for Microsoft's SharePoint or OBIEE.

I would take a look at JavaScript libraries like Sencha (ext-JS).
Check out their documentation: http://docs.sencha.com/ext-js/4-0/#!/api/Ext.grid.Panel
You can define a store with your data using the json format. You can add a field to your store based on the value of each record. If you need to have this value relative to all the others as percentile, you can listen to the 'load' event and traverse the records to calculate this 'high'/'low' value.
Ext.create('Ext.data.Store', {
storeId:'myStore',
fields:['destination', 'source', 'testName', 'testStatus', 'testStart', 'testEnd'],
data:{'items':[
// ... your data here
]},
proxy: {
type: 'memory',
reader: {
type: 'json',
root: 'items'
}
}
});
Then you can create a grid view of your data, using any of their built-in views. You can define Renderer that will add a class with the calculated value. Using simple CSS you can color each class with a different color.
Ext.create('Ext.grid.Panel', {
title: 'My Data',
store: Ext.data.StoreManager.lookup('myStore'),
columns: [
{ header: 'Test Name', dataIndex: 'testName' },
{ header: 'Value',
dataIndex: 'calculatedValue',
renderer: function(value) {
return Ext.String.format('<span class="{0}">{1}</span>', value, value);
}
},
// Other columns
],
height: 200,
width: 400,
renderTo: Ext.getBody()
});

This is exactly where d3js shines. There are few data visualization JS frameworks out there, you can look at KendoUI and sencha UI you can use as well, but just like google chart APIs they do not help you with complex math. D3JS does.

Related

are JSON schemas allowed to have custom fields? [duplicate]

This question already has an answer here:
Custom Properties in JSON Schema
(1 answer)
Closed 4 months ago.
language: {
type: "array",
items: {
type: "string",
enum: ["en", "es"],
},
ui_display: {"end": "English", "es": "Spanish"},
}
This is a JSON schema for an array. the "ui_display" field is obviously not a recognized property. Will anything break if I include it?
It's just a python dict/javascript object/whatever, so you can add properties as much as you like.
However, a lot of different things use JSON Schema. Some will be fine with this others will break I would think.
I know for example Connexion using OpenAPI will break, so there's at least one case which will break.
That being said, according to this answer (did you even google this before posting the question?) you can extend JSON Schema.

Error trying to reorder items within another list in Keystone 6

I'm using KeystoneJS v6. I'm trying to enable functionality which allow me to reorder the placement of images when used in another list. Currently i'm setting up the image list below, however I'm unable to set the defaultIsOrderable to true due to the error pasted.
KeystoneJS list:
Image: list({
fields: {
title: text({
validation: { isRequired: true },
isIndexed: 'unique',
isFilterable: true,
isOrderable: true,
}),
images: cloudinaryImage({
cloudinary: {
cloudName: process.env.CLOUDINARY_CLOUD_NAME,
apiKey: process.env.CLOUDINARY_API_KEY,
apiSecret: process.env.CLOUDINARY_API_SECRET,
folder: process.env.CLOUDINARY_API_FOLDER,
},
}),
},
defaultIsOrderable: true
}),
Error message:
The expected type comes from property 'defaultIsOrderable' which is declared here on type 'ListConfig<BaseListTypeInfo, BaseFields<BaseListTypeInfo>>'
Peeking at the definition of the field shows
defaultIsOrderable?: false | ((args: FilterOrderArgs<ListTypeInfo>) => MaybePromise<boolean>);
Looking at the schema API docs, the defaultIsOrderable lets you set:
[...] the default value to use for isOrderable for fields on this list.
You're trying to set this to true but, according to the relevant section of the field docs, the isOrderable field option already defaults to true.
I believe this is why the defaultIsOrderable type doesn't allow you to supply the true literal – doing so would be redundant.
So that explains the specific error your getting but I think you also may have misunderstood the purpose of the orderBy option.
The OrderBy Option
The field docs mention the two effects the field OrderBy option has:
If true (default), the GraphQL API and Admin UI will support ordering by this field.
Take, for example, your Image list above.
As the title field is "orderable", it is included in the list's orderBy GraphQL type (ImageOrderByInput).
When querying the list, you can order the results by the values in this field, like this:
query {
images (orderBy: [{ title: desc }]) {
id
title
images { publicUrl }
}
}
The GraphQL API docs have some details on this.
You can also use the field to order items when listing them in the Admin UI, either by clicking the column heading or selecting the field from the "sort" dropdown:
Note though, these features order items at runtime, by the values stored in orderable fields.
They don't allow an admin to "re-order" items in the Admin UI (unless you did so by changing the image titles in this case).
Specifying an Order
If you want to set the order of items within a list you'd need to store separate values in, for example, a displayOrder field like this:
Image: list({
fields: {
title: text({
validation: { isRequired: true },
isIndexed: 'unique',
isFilterable: true,
}),
displayOrder: integer(),
// ...
},
}),
Unfortunately Keystone doesn't yet give you a great way to manage this the Admin UI (ie. you can't "drag and drop" in the list view or anything like that). You need to edit each item individually to set the displayOrder values.
Ordering Within a Relationship
I notice your question says you're trying to "reorder the placement of images when used in another list" (emphasis mine).
In this case you're talking about relationships, which changes the problem somewhat. Some approaches are..
If the relationship is one-to-many, you can use the displayOrder: integer() solution shown above but the UX is worse again. You're still setting the order values against each item but not in the context of the relationship. However, querying based on these order values and setting them via the GraphQL API should be fairly straight forward.
If the relationship is many-to-many, it's similar but you can't store the "displayOrder" value in the Image list as any one image may be linked to multiple other items. You need to store the order info "with" the relationship itself. It's not trivial but my recent answer on storing additional values on a many-to-many relationship may point you in the right direction.
A third option is to not use the relationship field at all but to link items using the inline relationships functionality of the document field. This is a bit different to work with - easier to manage from the Admin UI but less powerful in GraphQL as you can't traverse the relationship as easily. However it does give you a way to manage a small, ordered set of related items in a many-to-many relationship.
You can save an ordered set of ids to a json field. This is similar to using a document field but a more manual.
Hopefully that clears up what's possible with the current "orderBy" functionality and relationship options. Which of these solutions is most appropriate depends heavily on the specifics of your project and use case.
Note too, there are plans to extend Keystone's functionality for sorting and reordering lists from both the DX and UX perspectives.
See "Sortable lists" on the Keystone roadmap.

How to achieve generic Audit.NET json data processing?

I am using Audit.Net library to log EntityFramework actions into a database (currently everything into one AuditEventLogs table, where the JsonData column stores the data in the following Json format:
{
"EventType":"MyDbContext:test_database",
"StartDate":"2021-06-24T12:11:59.4578873Z",
"EndDate":"2021-06-24T12:11:59.4862278Z",
"Duration":28,
"EntityFrameworkEvent":{
"Database":"test_database",
"Entries":[
{
"Table":"Offices",
"Name":"Office",
"Action":"Update",
"PrimaryKey":{
"Id":"40b5egc7-46ca-429b-86cb-3b0781d360c8"
},
"Changes":[
{
"ColumnName":"Address",
"OriginalValue":"test_address",
"NewValue":"test_address"
},
{
"ColumnName":"Contact",
"OriginalValue":"test_contact",
"NewValue":"test_contact"
},
{
"ColumnName":"Email",
"OriginalValue":"test_email",
"NewValue":"test_email2"
},
{
"ColumnName":"Name",
"OriginalValue":"test_name",
"NewValue":"test_name"
},
{
"ColumnName":"OfficeSector",
"OriginalValue":1,
"NewValue":1
},
{
"ColumnName":"PhoneNumber",
"OriginalValue":"test_phoneNumber",
"NewValue":"test_phoneNumber"
}
],
"ColumnValues":{
"Id":"40b5egc7-46ca-429b-86cb-3b0781d360c8",
"Address":"test_address",
"Contact":"test_contact",
"Email":"test_email2",
"Name":"test_name",
"OfficeSector":1,
"PhoneNumber":"test_phoneNumber"
},
"Valid":true
}
],
"Result":1,
"Success":true
}
}
Me and my team has a main aspect to achieve:
Being able to create a search page where administrators are able to tell
who changed
what did they change
when did the change happen
They can give a time period, to reduce the number of audit records, and the interesting part comes here:
There should be an input text field which should let them search in the values of the "ColumnValues" section.
The problems I encountered:
Even if I map the Json structure into relational rows, I am unable to search in every column, with keeping the genericity.
If I don't map, I could search in the Json string with LIKE mssql function but on the order of a few 100,000 records it takes an eternity for the query to finish so it is probably not the way.
Keeping the genericity would be important, so we don't need to modify the audit search page every time when we create or modify a new entity.
I only know MSSQL, but is it possible that storing the audit logs in a document oriented database like cosmosDB (or anything else, it was just an example) would solve my problem? Or can I reach the desired behaviour using relational database like MSSQL?
Looks like you're asking for an opinion, in that case I would strongly recommend a document oriented DB.
CosmosDB could be a great option since it supports SQL queries.
There is an extension to log to CosmosDB from Audit.NET: Audit.AzureCosmos
A sample query:
SELECT c.EventType, e.Table, e.Action, ch.ColumnName, ch.OriginalValue, ch.NewValue
FROM c
JOIN e IN c.EntityFrameworkEvent.Entries
JOIN ch IN e.Changes
WHERE ch.ColumnName = "Address" AND ch.OriginalValue = "test_address"
Here is a nice post with lot of examples of complex SQL queries on CosmosDB

How would you design this database using MongoDB? Or would you use a different kind of database? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
I would like to use MongoDB for my next web project. Being only familiar with SQL databases so far, I'm wondering if this design makes sense for MongoDB and if it's scalable, or if there's a better way to do it:
Basically I will have a LOT of users who will all follow the same course to study vocabulary on my site. For each user, I need to keep track of the words he has studied (between 0 and 2000, probably 100 on average), how well he knows them and when he will have to review them again. I'd also like to keep track of the next few words coming up that he should learn for ease of reference, and some other details about the user. I thought of a design like this:
{
"user" : "Judith",
"country" : "Germany",
"curr_unit" : 1,
"curr_lesson" : 2,
"words" :
{
"我" : { "state" : 7, "next_review" : "2013-09-21 19:43:37 +0100" },
"你" : { "state" : 7, "next_review" : "2013-09-21 19:43:37 +0100" },
"是" : { "state" : 7, "next_review" : "2013-09-21 19:43:37 +0100" },
"学生" : { "state" : 2, "next_review" : "2013-08-31 19:43:37 +0100" },
"医生" : { "state" : 1, "next_review" : "now" },
"吗" : { "state" : 1, "next_review" : "now" },
... (could be 0-2000 such items) ...
},
"next_words" : [ "他", "不", "她", "中国人", "美国人", "这", "老师", "好", "很", "谢谢" ],
}
Basically, I'm having doubts that this is how MongoDB is intended to be used because I can't seem to retrieve words by state, I can only retrieve all information about the user at once. Also, it's annoying having to treat timestamps as strings.
Should I be using an SQL database instead? I'm afraid that the vocabulary_users table would quickly get unmanageable with lots of users and on average 100 entries each.
As this is a spaced repetition system, most of the times I will access this document in order to figure out which words should be studied next (using a mix of "state" and "next_review"). The second most common type of access will be when either "state" or "next_review" have to be updated after a study session. The third most common type of access is updating "curr_lesson". Everything else occurs less than 10 times over the lifetime of the user.
Doesn't seem like making a lot of sense. First, why have you decided to make "words" an object? If anything, it should be an array. It seems like you are building some sort of spaced repetition system rather then a dictionary, so instead of answering queries like: what do we know about word "德国人", you rather want to answer question: which card has highest priority to be shown now. So, if you make "words" an array of objects in form of { "word" : "数据库", "state" : 7, "next_review" : "2013-09-21 19:43:37 +0100" }, you can build an index on {"user":1, "words.next_review:1}.
Next, it seems like embedding "words", and maybe "next_words" as well, into the user document is not really good idea. Your "words" will constantly be updated, which will cause the whole user document to be rewritten. If the document doesn't fit into its old space, it will be reallocated, which in turn means that all indexes for this collection will have to be updated.
Last piece I want to add is that almost half of the space in your document is taken by keys. Keys are something that is repeated in each document and it's a good practice to make them very short.
You can almost certainly make this work in SQL, in MongoDB, or in some other kind of data store like Redis, or Lux. If your interest is in learning MongoDB, then use MongoDB and figure out the solutions to your problems! but if you just want to get the project working quickly and have easy access to a lot of tools, following a well-paved path, you should probably stick with Postgres or some other SQL database. That's my advice.

Creating a custom defect trend chart using App SDK 2.0

I've been tasked with plotting some defect charts using Rally historical data. Right now I'm using a simple REST client to pull data at certain points in time and plot the count on a spreadsheet. What I'm doing right now is:
{
find : {
"_ProjectHierarchy": <projectId>,
"_TypeHierarchy": -51006,
"FoundInBuild" : {$regex: "3\\.3\\."},
"State" : {$in : ["Submitted","Open"] },
$or: [
{"Severity" : { $in : ["Catastrophic","Severe"] }},
{"Priority" : "showstopper"}
],
"__At" : "<date>"
},
pagesize : 1,false
}
I just run this once for every date I need the data for. That's a lot of queries! What I'm looking for is a way to run a single query using _ValidFrom and _ValidTo to enclose a time range, then pass it on to a SnapshotStore, then plot that on a Chart? I'm certain there's a way to do it, but I can't figure it out from the docs. Any help much appreciated.
Unfortunately, the example space for AppSDK2 and Lookback API is presently a bit thin. There are some cool apps out there, for instance, you may wish to check out David Thomas' cool Hackathon app:
Defect Re-work Trend
As a starting point. It queries LBAPI for Defects and stores the resulting data in a SnapshotStore. The App itself measures Defect "thrash" or the trending around how many times a Defect is re-opened during a particular development cycle.
In reviewing Hackathon apps, just be aware that certain methods and syntax for the SnapshotStore may change slightly in a future release of AppSDK2.