I have statistics for a project stored in a MySQL database and looking for a quick way to prototype a webpage which graphs various relationships in the data, using bar graph, line graph, pie chart, etc. I found something called "Dashing", but seems to use something called "coffeescript" which I'd rather not learn in my short time frame (~1 week).
Does anyone know of any good packages/frameworks that can help out with the job? If I could do it using only C/C++/obj-C it would be ideal, though Java is possible as well.
I decided on using PHP/Apache/MySQL and the Jpgraph PHP graphing framework.
Related
As my question states, I am building an application that has a feature of displaying analytics for the user. How many units sold, product price breakdowns, etc.
My initial thought was to use MongoDB to test it out, with mongoose as the ORM. I knew I would need some relational functionality that MongoDB handles good enough so at the end I'm still happy with MongoDB from a dev perspective but whenever I start thinking about building charts, data queries, and analytics, things start to get a little bit muddy since I'm not entirely sure how to go about it.
Right now whenever I need to build a chart I run a bunch loops over data to format the relevant information for the front end to read. However, this is probably inefficient and additional server logic that could be handled differently with say a SQL database.
Having given the context, my question is: Is mongodb not a good database for what I want to do? If so, what would be your recommendation? Or is my inexperience with mongo making me think about changing technologies without looking at alternatives? If so what should I be educating myself on?
Have written an application in Python that handles a large amount of data used to generate graphics. I need help finding a technique to allow the user to scroll through the data, in both directions, while displaying a segment of the data in a graphic. Ideally only data which would be visible would be read from the database.
I am currently using a combination of Pandas, Sqlalchemy and bqplot, but if there are better packages for implementing the desired functionality I am willing to change.
Sqlalchemy seems like a good bet for handling the database but I have found the documentation hard to understand and need a nudge in the right direction. Any help or advice will be appreciated.
Thanks,
Steve
You can store your data in a grid whose each rectangle is the size of the user's display:
This way, you need to load at most 4 rectangles.
As to the database, sqlalchemy is great, but it does fancy ORM stuff you don't need. In fact, as a rule of thumb, if you're not going to do table joins (which, by your description you're not), you don't need an SQL database at all, but rather a large key-value store. Here is a pymongo tutorial, for example, but just choose any key-value store that works for you.
I am working on a project which involves collecting dynamic form data. These forms are user-defined (think surveymonkey) and thus a fixed schema cannot be defined for them. Data in terms of questions/answers would be retrieved for these forms and then stored into the database. Reporting/Searching on this answers (filtering and aggregation) is of utmost importance. There are two approaches which are feasible.
Use a SQL database and store the each field data as a separate row. Reporting/Searching is then done via SQL. My apprehension is that it would result in complicated joins for reporting.
Use a NoSQL database like MongoDB. This seems to be a perfect fit for storing the dynamic data since it is schema-less. However, I am not sure how good its reporting capabilities are.
It seems easier for target users to learn sql than to define map/reduce queries. How easy would it be to build a UI for reporting/searching over mongoDB.
Simple things like - list of users who gave a particular set of answers. How many such users over a period of time etc?
Thanks,
Pulkit
It's already been mentioned in the comments, but I'll re-iterate that you should look at Mongo's map/reduce functionality for reporting and the aggregation framework.
Having done map/reduce in both Couch and Mongo I can say that they are very similar. It's definitely a barrier to entry for a developer that isn't familiar with it, but once you get a few working examples, it's not too bad.
Consider that Mongo can output a map/reduce job to a collection, which I've found to be really useful. This means you can schedule the jobs and run them periodically and output to a place that you can then report on. It's not that hard to create a framework that lets developers write simple Javascript map and reduce functions and then plug them in to be run on a schedule.
The aggregation framework is much easier to understand for a developer coming from SQL. Still a learning curve, but not as bad as map/reduce. It is much more well suited to ad-hoc reporting queries and there is nothing comparable in Couch.
You could maybe make a reporting UI that maps to the aggregation framework, but I wouldn't try to do something similar for map/reduce queries.
I would like to know what I have to think about in order to download an SQL DB into core data? I am not sure what frameworks I would have to use or if there are any particular requirements when formatting the SQL DB.
Any help, suggestions, links to tutorials would be hugely appreciated.. I have done some searching around and its just hard to make sense of things because I am not sure if what I am looking for is even correct.
Im not sure if I understand what you are after, but maybe 2 links would be helpful:
an application that lets you load you data to a Sqlite DB from a lots of formats (Excel, xml, ...) is Navicat. It's an easy way of getting data to the DB. As David H mentioned, you could then work with the data without using Core Data via an Sqlite wrapper like FMDB. Then you can access the data with SQL commands.
A totally different tutorial is offered by nsscreencst.com: Importing into Core Data is a tutorial that shows how you can import JSON data to Core Data from a web API. This might be related to your use case. Unfortunately the videos there cost $9/month, but IMHO they are doing a terrific job.
(I'm not affiliated with either of the above companies)
This is just not possible - even though Core Data can use SQLite for storage there is no import/export. You really have two options:
1) If you have one database you want to use Core Data with within an iOS app, then you can write a really simple Mac App that interacts with your SQL store, and essentially replicates it in Core Data using a Core Data Schema you create based on your SQL database. The advantage of this is that is simpler to test and develop using a mac app. The final Core Data repository can then be used by your iOS app (by including the Core Data Schema with it).
2) Do all the import in your iOS app. This may take longer to develop but you can then dynamically download the SQL database into the phone and use SQLite to read it.
To import your data you have to create NSManagedObjects objects from every single record in your SQLite database. Depending on the complexity of your database model this can be very tricky.
There is a good introduction by Marcus Zarra on how to create NSManagedObjects from JSON: https://stackoverflow.com/a/2363996/480069. This should give you an idea on how to start. As Marcus said be aware of the relationships in your object graph so you don't end up in a loop when having complex relationships.
There are also a lot of good tutorials out there how to serialize NSManagedObjects, so just give Google a try: NSManagedObject serialize.
So I'm having a head against the wall moment and hoping somebody can come help either remove the wall or stop my head from moving!!
Over the last 3/4 weeks I've been investigating ORM's in readyness for a new project. The ORM must map to an existing, large and ageing SQL database.
So I tried Subsonic. I really liked v2 and v3 after modding to work nicely with VB and named schemas in SQL was running OK. However, its lack of flexibility of having separate entity properties names vs column names had me pulling my hair out (sorry Rob).
I tried Entity Framework but I found like others it lacking in certain areas.
So I bit the bullet and tried nHibernate but after a week or so getting it working how I liked (with help from Codesmith to generate classes/hbms for me) I'm frustrated with the time it takes to startup (build a config object), despite trying a number of tricks to reduce this time.
I'm essentially after building a DAL class that I can share between apps and websites. Am I barking up the wrong tree? For a legacy project with 100s of tables should I go back to ado.net and use DTOs? Aarrgh!
Sorry for the ranty style of question. I don't have much hair left and I'd like to keep what I have!!
Thanks in advance, Ed
PS. I should add that I know SQL very well and not scared of getting my hands dirty to write fast queries. If anything I don't need to be hid from SQL
ORM let's you:
To map table rows to objects, that are the the workable pieces of object oriented programming.
To automatically navigate through object relationships
To easily add, edit and remove table rows
To query the database in a more intuitive way as you don't have to think of joins (this one will depend on the ORM and the query method)
To transparently handle L1 and L2 cache.
All of the above would have to be handled by hand if you werent using ORM.
PS: I agree to Dmitry as to the startup time of NHibernate (see question comments). Besides, did you try Fluent NHibernate? Fluent NHibernate is impressively easy. I couldn't believe my eyes when I first mapped a database. It's even easier than proprietary ORMs like DevExpress XPO.
The biggest benefit of an ORM tool is that it will help you layer your application correctly. Most project nowadays use a Data Layer to connect to the database. You start from the ORM tool to produce classes that correspond to your database objects. Then you define an interface using these methods. All persistence code uses the methods of this interface. This way the business logic layer is only coupled to this higher-layer interface and needs to know nothing about the database. In fact there should be no dependency on ADO.NET or even NHibernate.
Another advantage of ORM tools is that you de-couple your application from the database server. You could change the db engine and still use the same code. Also there isn't only the complexity of the SQL that the ORM hides from you. It can also help you with transactions logic and connection pooling.
I'd say that for new projects an ORM tool is a necessity. For legacy projects it isn't so much beneficial, unless of course you have the time/money to start from scratch.
In my experience, most ORMs end up being way more complex than SQL. Which defeats the entire purpose of using them.
One solution I'm enthusiastic about is LINQ2SQL. It excels as a thin layer about stored procedures or views. It's really easy to use and doesn't try to hide SQL.
There are basically two questions here:
What's great about ORMs? There are similar questions on Stackoverflow. See:
What are the advantages of using an ORM?
Is everyone here jumping on the ORM band wagon?
How can I improve NHibernate startup time? See:
http://ayende.com/Blog/archive/2007/10/26/Real-World-NHibernate-Reducing-startup-times-for-large-amount-of.aspx
http://nhforge.org/blogs/nhibernate/archive/2009/03/13/an-improvement-on-sessionfactory-initialization.aspx