Looking for some advice in displaying reports with Google charts (for now table reports, but will go on to other types later)
Are people creating a view for each specific report? Or is it better to re-use an existing view and dynamically create the table from it.
I've found some resources where the table columns are dynamically generated, but I am not quite sure how to check and specify the column to be a string or numeric data type.
If I create views for each report, I know ahead of time what the table structure will be and thus making it easier to create, but perhaps more of a nightmare to maintain down the road.
Any thoughts?
Depending on how often the columns/tables change, one of these will work for you.
Create 10 Views
If your data doesn't change too often, just create 10 different DataViews based on the 10 tables you want to create. Since they all use the same underlying data, any change to the original DataTable will automatically be reflected in all 10 tables, and the performance will be much quicker than creating 10 different tables.
If you change the views often, maintenance may be a hassle, so I would suggest a different approach.
Create a DataView Creation Function
You can create a function that will take an array of columns and create a DataView from those columns. This way you can change the views easily by changing the function call only, and will make your code look simpler. Depending on how you want to set the DataView (whether you use filtered rows and/or columns) the function will be slightly different, but this will allow you the most flexibility to change things while giving all the benefits of having 10 dataviews (updating based on changes to the datatable, etc.).
Automate the Whole Kit and Caboose
If you're going to go that far, you can just automate the entire process. Select the rows/columns you want from the DataTable, give a <div> name (or create a <div> automatically), then create the view and the new table chart drawing function. This will work great if your tables all behave the same (have the same events, options, etc.) as it is going to be of use far in to the future.
On the other side, it is more work than the above two to initially create.
Related
We are having a system that gather large quantities of data each month and performes rather advanced calculations that increase the database even more. Since we have the requirements from the customer that we need to store data for fast access three years back and that we must be able to access older data (up to ten years), this however can be low performance and requires some work. We want to avoid performance issues where the database and its tables grows out of proportion.
After discussing using SQL Enterprise (VERY costly and full of traps since we haven't gotten the know-how) and since our system have so many tables that referenses each other we are leaning towards creating some kind of history tables to which we move data in a monthly fashion and rewrite the select queries that we have based on parameters to search either in the regular table or in the history or both depending on the situation.
Since we also are using NHibernate for mapping I was wondering if it is possbible to create a mapping file that handles this by itself (almost) using some sort of polymorfism or inheritance in which each object is stored in different tables based on parameters?
I know this sounds complicated and strange and that there is other methods to perform this but I this question I would rather have people answering the question asked and not give other sugestions to use instead.
As far as I know NHibernate can't do that (each class can be mapped to one table/view )but you can use SQL Queries or StoredProcedures (depends on the version of NHibernate that you are using) to populate mapped objects.
In your case you can have a combined view created by making unions of different tables Then you can use a SQL query to populated your entity.
There's also another solution that you create a summary object for your queries that uses that view ,therefore you can use both HQL and criteria to query this object.
Short answer "no". I would not create views as you mention a lot of joining.
Personally I would create summary tables and map to these directly using a stateless session or a very least mutable=false on the class definition. Think of these summary tables as denormalised data for report only. The only drawback is if historic data changes on a regular basis then the summary tables also needs changing. If historical data never changes then this should be simple to achieve.
I would also most probably store these summary tables on another catalog rather than adding to the size of the current system.
Its not a quick win this one I am afraid.
I have a few challenges I need help on. I need to pull data in to my SQL database from arbitrary sources.
The details are: I know the exact structure of my database and the structure will not change. When I do take in new data, it will occur only one time, at the time I set up an instance of my database. I will make many instances of my database and each time it will have to pull data from a different source, and those sources will be structured in different ways.
The data will most likely contain thousands of rows of records. The data source will most likely be held in Excel, Access, more rare Word and even rarer, it'll be held in a SQL database.
I can assume that most of the core data will be the same, just put in different locations. They will follow a general grouping despite how there held.
Essentially, I'm transferring data from legacy systems to a SQL system and this must be done for many groups and they need their own private instance of the database.
Any thoughts on how I would do this? How hard would it be to write a program that would do most of this for me?
This is definitely a real-world question. Is it possible to write a program that will do most of this? Not most of this, I think, but perhaps some of it.
For each table in your target system, create a view that displays the source data you expect to be able to insert. Choose column names that make it easy to tell what has to be done; most likely you'll choose column names that match the target columns in your INSERT statement. Save your INSERT statements as stored procedures.
Now, when you are given a new source of data in a new format, you will still have to recreate your views, but once the views are displaying the right data under your chosen column names, you can run your stored procedures without change.
I have a similar type of project where data is being retrieved from Access, .ini file, file modification dates, and MySql. I scrape this data every morning and basically append to a set SqlServer schema.
I created a DataTable and as I iterate a set of directories, insert the data into each new row. Once I have the DataTable complete, I perform a bulkcopy to append to the database.
I hope that helps you out a bit. I know my project doesn't cover all the aspects of your question; but also don't have a DBA to provide views, stored procedures, etc. Nor do I have the additional time to devote to such things. Not the most favorable of conditions, but that's the way it is.
HTH...
The best way of solving this problem is with and ETL (Extract-Transform-Load) solution. A good choice is SSIS which is through Microsoft's BI suite.
This is the building blocks for consciousness or the base......
1 A data base that organizes thousands of files similar to dna,
2 user interface
3 parts are hidden, preventing a system breach/crash
Recently I faced an interview and I was asked the above question.
I was dumb when I think about it.
Interviewer said:
All people are saying views have lots
of advantages but I find no
disadvantages, why so?
when table is not there view will not work.
dml is not possible if that is more than one table.
it is also database object so it will occupy the space.
When table is dropped view becomes inactive.. it depends on the table objects.
Querying from view takes more time than directly querying from the table
Most of the things I would say have already been covered, I would add this though.
Views are useful in many situations but making too much use of them can be a mistake because they tie your hands in terms of query structure. Often when your overall query contains several views within it (especially when views are layered), or when a view has been adapted for a slightly different purpose to what was originally intended, you find that there is a far better way of writing the query if you just expand the views and change the logic.
Like any tool, views can be misused particularly when you're not sure how they should be used properly.
Chris Mullins defines three basic view implementation rules:
The View Usage Rule
The Proliferation Avoidance Rule
The View Synchronization Rule
If you don't get these things right you get code maintenance problems, performance problems, security problems, etc.
The only disadvantage I can think of is that you may force the user to join several views to get the data in a way that is useful to them, as you now have largely static queries.
So, if the view was created one time and it is expected to never change, you may end up with a preponderance of views that creates a maze for the user to navigate through, so there should be some process to update views, to keep them useful as needs change.
1) when a table is dropped ,view will be affected.
2) If column name is renamed then view will show exception "Invalid column name" .
3)When view is created for large table ,it occupies some memory .
If you write some complex views, while querying simple data from view it will take more time.
It affects performance. Querying from view takes more time than directly querying from the table.
If view would join more than one table, you may not perform any DML
operations.
Table dependence- if you change table, you need to
updated view also.
A view permits the DBA (database administrator) to tightly control what goes in and comes out of a database.
In banking a view is often used to permanently keep track of every change made to the table. The real table typically contains additional columns that are not seen by "the view" such as:
last-modified (when the last change was made)
last-action (update/delete/add)
last-actioner (person who updated the row)
So when displaying the view of the table only the latest update or add of any row is displayed. However the table still contains every existing change and row deletion.
The major downside to a view is to the user of the table (the application programmer) who cannot directly change the underlying table (for performance reasons, for example). Additionally it does create more work for the database administrator. You might also consider the extra CPU burden placed upon the server - particularly if it is utilised by many clients.
I've shown up at a new job and discovered database which is in dire need of some help. There are many many things wrong with it, including
No foreign keys...anywhere. They're faked by using ints and managing the relationship in code.
Practically every field can be NULL, which isn't really true
Naming conventions for tables and columns are practically non-existent
Varchars which are storing concatenated strings of relational information
Folks can argue, "It works", which it is. But moving forward, it's a total pain to manage all of this with code and opens us up to bugs IMO. Basically, the DB is being used as a flat file since it's not doing a whole lot of work.
I want to fix this. The issues I see now are:
We have a lot of data (migration, possibly tricky)
All of the DB logic is in code (with migration comes big code changes)
I'm also tempted to do something "radical" like moving to a schema-free DB.
What are some good strategies when faced with an existing DB built upon a poorly designed schema?
Enforce Foreign Keys: If a relationship exists in the domain, then it should have a Foreign Key.
Renaming existing tables/columns is fraught with danger, especially if there are many systems accessing the Database directly. Gotchas include tasks that run only periodically; these are often missed.
Of Interest: Scott Ambler's article: Introduction To Database Refactoring
and Catalog of Database Refactorings
Views are commonly used to transition between changing data models because of the encapsulation. A view looks like a table, but does not exist as a finite object in the database - you can change what column is being returned for a given column alias as desired. This allows you to setup your codebase to use a view, so you can move from the old table structure to the new one without the application needing to be updated. But it means the view has to return the data in the existing format. For example - your current data model has:
SELECT t.column --a list of concatenated strings, assuming comma separated
FROM TABLE t
...so the first version of the view would be the query above, but once you created the new table that uses 3NF, the query for the view would use:
SELECT GROUP_CONCAT(t.column SEPARATOR ',')
FROM NEW_TABLE t
...and the application code would never know that anything changed.
The problem with MySQL is that the view support is limited - you can't use variables within it, nor can they have subqueries.
The reality to the changes you wish to make is effectively rewriting the application from the ground up. Moving logic from the codebase into the data model will drastically change how the application gets the data. Model-View-Controller (MVC) is ideal to implement with changes like these, to minimize the cost of future changes like these.
I'd say leave it alone until you really understand it. Then make sure you don't start with one of the Things You Should Never Do.
Read Scott Ambler's book on Refactoring Databases. It covers a good many techniques for how to go about improving a database - including the transitional measures needed to allow both old and new programs to work with the changing design.
Create a completely new schema and make sure that it is fully normalized and contains any unique, check and not null constraints etc that are required and that appropriate data types are used.
Prepopulate each table that fills the parent role in a foreign key relationship with a single 'Unknown' record.
Create an ETL (Extract Transform Load) process (I can recommend SSIS (SQL Server Integration Services) but there are plenty of others) that you can use to refill the new schema from the existing one on a regular basis. Use the 'Unknown' record as the parent of any orphaned records - there will be plenty ;). You will need to put some thought into how you will consolidate duplicate records - this will probably need to be on a case by case basis.
Use as many iterations as are necessary to refine your new schema (ensure that the ETL Process is maintained and run regularly).
Create views over the new schema that match the existing schema as closely as possible.
Incrementally modify any clients to use the new schema making temporary use of the views where necessary. You should be able to gradually turn off parts of the ETL process and eventually disable it completely.
First see how bad the code is related to the DB if it is all mixed in no DAO layer you shouldn't think about a rewrite but if there is a DAO layer then it would be time to rewrite that layer and DB along with it. If possible make the migration tool based on using the two DAOs.
But my guess is there is no DAO so you need to find what areas of the code you are going to be changing and what parts of the DB that relates to hopefully you can cut it up into smaller parts that can be updated as you maintain. Biggest deal is to get FKs in there and start checking for proper indexes there is a good chance they aren't being done correctly.
I wouldn't worry too much about naming until the rest of the db is under control. As for the NULLs if the program chokes on a value being NULL don't let it be NULL but if the program can handle it I wouldn't worry about it at this point in the future if it is doing a default value move that to the DB but that is way down the line from the sound of things.
Do something about the Varchars sooner rather then later. If anything make that the first pure background fix to the program.
The other thing to do is estimate the effort of each areas change and then add that price to the cost of new development on that section of code. That way you can fix the parts as you add new features.
How would you create a database in Microsoft Access that is searchable only by certain fields and controlled by only a few (necessary) text boxes and check boxes on a form so it is easy to use - no difficult queries?
Example:
You have several text boxes and several corresponding check boxes on a form, and when the check box next to the text box is checked, the text box is enabled and you can then search by what is entered into said text box
(Actually I already know this, just playing stackoverflow jeopardy, where I ask a question I know the answer just to increase the world's coding knowledge! answer coming in about 5 mins)
My own solution is to add a "filter" control in the header part of the form for each of the columns I want to be able to filter on (usually all ...). Each time such a "filter" control is updated, a procedure will run to update the active filter of the form, using the "BuildCriteria" function available in Access VBA.
Thus, When I type "*cable*" in the "filter" at the top of the Purchase Order Description column, the "WHERE PODescription IS LIKE "*cable*" is automatically added to the MyForm.filter property ....
Some would object that filtering record source made of multiple underlying tables can become very tricky. That's right. So the best solution is according to me to always (I mean it!) use a flat table or a view ("SELECT" query in Access) as a record source for a form. This will make your life a lot easier!
Once you're convinced of this, you can even think of a small module that will automate the addition of "filter" controls and related procedures to your forms. You'll be on the right way for a real user-friendly client interface.
This is actually a pretty large topic, and fraught with all kinds of potential problems. Most intermediate to advanced books on Access will have some kind of section discussing "Query by Form," where you have an unbound form that allows the user to choose certain criteria, and that when executed, writes on-the-fly SQL to return the matching data.
In anything but a flat, single-table data structure, this is not a trivial task because the FROM clause of the SQL is dependent on the tables queried in the WHERE clause.
A few examples of some QBF forms from apps I've created for clients:
Querying 4 underlying tables
Querying a flat single table
Querying 3 underlying tables
Querying 6 underlying tables
Querying 2 underlying tables
The first one is driven by a class module that has properties that reflect the criteria selected in this form, and that has methods that write the FROM and WHERE clauses. This makes it extremely easy to add other fields (as long as those fields don't come from tables other than the ones already included).
The most complex part of the process is writing the FROM clause, as you have to have appropriate join types and include only the tables that are either in the SELECT clause or the WHERE clause. If you include anything else, you'll slow down your query a lot (especially if you have any outer joins).
But this is a big subject, and there is no magic bullet solution -- instead, something like this has to be created for each particular application. It's also important that you test it thoroughly with users, since what is completely clear and understandable to you, the developer, is often pretty darned mystifying to end users.
But that's a principle that doesn't just apply to QBF!
At start-up, you need to show a form and disable other menus etc. That way your user only ever sees your limited functionality and cannot directly open the tables etc.
This book excerpt, Real World Microsoft Access Database Protection and Security, should be enlightening.
For a question that vague, all that I can answer is open MS Access, and click the mouse a few times.
On second thought:
Use the "WhereCondition" argument of the "OpenForm" method
If the functionality is very limited and/or specialised then a SQL database is probably going to be overkill anyhow e.g. cache all combinations of the data locally, in memory even, and show one according to the checkboxes on the form. Previously you could have revoked permissions from the table and granted them only on VIEWs/PROCs that queried the data in the prescribed way, however security has been removed from MS Access 2007 so you can you now really stop users bypassing your simple app using, say, Excel and querying the data any way they like ...but then isn't that the point of an enterprise database? ;-)