Even with the advanced technologies and available database tools (even free alternatives) are available today, it seems that huge number of users are still very comfortable in using Excel IN EVERYTHING! That's why, as a database developer working as one of these users, I am forced to let them use Excel simply because they are very comfortable using it. Especially for the older people who seemed to never gonna let Excel go and embrace a new tool.
Currently, to make their experience as smooth as possible and at the same time, automated, I'm using a lot of database queries inside Excel be it view,SQL or stored procedures. Mostly on ad-hoc (but then became permanent) reports. My question is are there any hopes to improve this situation? I'm sure a lot of organizations are using this same method. Is it possible to completely replace this arrangement with something more logical and efficient both in data collection and reliability? I'm thinking about using Sharepoint. Am I on the right track?
I have also struggled with this problem in the past and can say that what worked for me was a two pronged approach.
Step 1 – Make a good alternative
It sounds like you have already done this, depending on the system there will always be some random report that someone needs to run to suit their “Business Need”. There is no way that you could cram all of these into your system as it would fill up with reports and the users would become snow blind.
Step 2 – User education
Show them the new way of making their own reports (Business objects SSRS whatever) and make sure they are comfortable with it. This is the hardest part as some people like their comfort blanket of excel and wont want to leave it. Give them some templates and some standard reports, maybe even pair develop one or two reports at their desk with them so they get the knack of it.
I will leave on a bit of a daily WTF, there was once this expert business manager who was an expert in business objects. She made reports left right and centre but she treated it like a giant version of excel and her work was littered with examples of this i.e. one report she wrote was to get the dealing totals for a year. No problem I hear you cry just do
SELECT SUM(DealAmount) where DealDate Between X and Y
Nope not our business expert, in here excel frame of mind this was too much like black magic so what she did was return a row for EVERY SINGLE DEAL done in that year and then aggregated it client side to give her a total. In I step and wow the users by reducing this 104mb report that took 17 minutes to run down to a 100kb report that ran in about 15 seconds.
I would go the other way around. And I mean by that, not making queries and database connections within Excel, but using some sort of Web Application to let users (through wizards) generate data they need, and export them to Excel to work.
That way you will have the following benefits:
No DB connections (and probably passwords) in your Excel files
No distribution problem of Excel files with new queries, views, etc.
Centralized approach to data retrieval
Excel for users used to it
Back in the day, I loved using Crystal Reports for ad-hoc reporting. I'm not sure about it's current status, as it seems that SAP has purchased the product: http://www.sap.com/solutions/sapbusinessobjects/sap-crystal-solutions/index.epx
Related
I am working using Tableau and have to write down multiple different SQL each time, while making new data sources.
I have to save all changes on SQL for every data source.
Currently I would paste the SQL on notepad and save them on separate folder in my computer, along with description of the changes.
Is there any better way to do this?
Assuming you have permission to create objects in the database, begin by creating database views, As #Nick.McDermaid commented.
Then, instead of using Custom SQL data source in Tableau, just connect to the View as if it were a table.
If you need to track the changes to these SQL views of your data, you will need to learn how to use source control for the .sql files that can be scripted from within SQL Server Management Studio:
Your company or school may have a preferred source control system already in use, in which case you should use that. If they don't, or if you are learning at home, then Git and Subversion are popular open source choices.
There are many courses available on learning platforms like Coursera that will teach you how to learn how to use those systems.
I had similar problem as you.
We ended up writing the queries in SQL Editor SQL Work bench (https://www.sql-workbench.eu/), then managed the code history and performed code peer-review (logic, error check, etc) in team shared space (like confluence).
The reasons we did that is
1) SQL queries are much easy to write on Work Bench
2) Code review is a must! You will find through implementing a review process more mistakes than you could ever think about
3) The shared space is just really convenient as it is accessible by everyone, and all errors are documented. After sometimes you get a lot of visible knowledge accumulated.
I also totally agree with Nick as this is one step to a reporting solution. But developing a whole reporting server is heavy, costly and takes time. Unless management are really convinced of the importance of developing a reporting solution, you may have to get a workaround with queries and Tableau (at least that was the case for us)
A little late to the party, but I would suggest you simply version the tableau workbook. The contents of the workbook are XML, so perfect for versioning using file based tools (Dropbox, One Drive, etc.) or source control (git, etc.). The workbooks themselves are usually quite small, so just make sure to keep the extract data separate if you use it.
As a way to score points for the study I’m doing, as well as out of interest into databases and wanting to help my team I’m trying to build a capacity planning tool in MS Access 2007. I work in a department that handles registering and supporting tenders. I have attached two pictures of what I’m trying to do here.
I’ve already spent some weeks making multiple iterations with colleagues who are involved and help write VBA and SQL (out of interest, wanting to learn something or otherwise. Our core business, however, isn’t developing). The primary goal of the database is as follows:
A user can access, create and modify “cases” that correlates to a case ID that we use in a different system.
A user can write down his capacity per week per year for a case.
multiple users can assign themselves to a case.
Users can leave messages (records) for other users to see on a case
Metadata can be attached to the case
The main problem we seem to be running into is that whenever a user tries through to edit an existing case through the overview, the case data no longer “complies” with entries elsewhere. Forcing updates through visual basic also seems to not have worked so far.
Adding to the complexity: most of the names we use are in dutch.
Here is an overview of the relations.
http://imgur.com/O022LAG
Here is a screenshot of the case overview as seen by a user.
http://imgur.com/kuENqaq
Main question:
How can I make entire records change for multiple users based on the input of one user.
In compliance with the guidelines regarding asking subjective questions I’m trying to be a bit more precise here:
Additionally I’m uncertain:
whether it is our approach that is wrong,
if perhaps we’re overlooking a glaring issue, or
if we should redesign this from scratch with a different layout.
Any help specifying where we should look or what would be advisable to do would be much appreciated!
Kind regards,
Timo
Why excel? Well excel is what is used to import the player salaries.
Now I need the spreadsheet to do the following.
Create teams within salary cap.
Include/exclude specific player function
Build multiple lineups from a selected list of players within the cap
Can I do all of this with excel? or do I need to know excel vba as well?
Also which parts of excel or if necessary excel vba must I need to know to code such a thing? Also if someone could give me a short summary of the steps needed to hypothetically make such a thing it would be great. Thanks.
I'm posting this as a reply because it is too long for a comment window.
Just because Excel has a grid doesn't mean that it is fit for data storage and data handling, on the contrary.
What you typically want to do is create a transparent structure that guarantees the integrity of your data and that allows dynamic portability when needed one day.
Excel is meant to be a spreadsheet, people forget this all the time or they just avoid the topic: although Excel has a grid, doesn't mean that it is a good fit for reliable data storage. It is not even the least complex way of storing data, depending on the amount of VBA that you need to manage all these data and the gates that you unnecessarily open towards potential bugs.
This is why an RDBMS is what will fit your needs, in this case Access would be a good option as it preserves your data integrity if you get the table structure right and it executes a lot of tasks for you that you should otherwise need to program yourself to protect the integrity of your data.
Although you can perform SQL on spreadsheets too, note that Excel does NOT cover related tables (what you typically seem to need for building your teams and salary limits), so what many Excel programmers will typically do is to create their own code to make this cross-table data storage thing work.
Don't do this if the alternative is available and much more reliable and future-proof.
At first sight, it seems that you won't even need any VBA; I'm not sure of all the things you want to do, but my first impression is that you can manage everything with SQL syntax and stored queries in MS Access. You can import Excel sheets into Access if you get their format right so that should not be a problem.
Once your data is stored there, rest assured that you have made your life a lot easier.
I need to query a database and save over six million records for later analysis and reports using excel VBA. What would be the best way to do this ? I need to be able to access the data later, analyze it and generate reports on analyzed data ( I should be able to access the whole data locally in my system and should be able to process it fast) How good is MS access to save data if the amount of records run in to millions? I do not have a lot of experience using oracle and VBA together. Is it a good way to go if used with excel VBA? Appreciate it if anyone could give some links which might help me in the task.
Note: I don't have a choice to use another language, due to reasons beyond my control. I am reasonably good in Java, but can't use it.
If you want to manipulate more than 1 million rows of data in Excel, you should look into PowerPivot, which is an add-in for Excel 2010 (it's built into Excel 2013). It allows 100 million rows of data in addition to calculated fields on each row, although these calculations will likely be slower than the SQL backend / Excel frontend solution others have recommended.
I need to query a database and save over six million records for later analysis and reports using excel VBA.
I would hope that the analysis will be done on a SUBSET of those records given that even with Excel 2010 you're limited to 1,048,576 rows.
I need to be able to access the data later, analyze it and generate reports on analyzed data ( I should be able to access the whole data locally in my system and should be able to process it fast) How good is MS access to save data if the amount of records run in to millions?
It'll do it. I've done it (including, many years ago, via a direct connection to an Oracle database), though the joker in the pack is the question of how many columns are in there as one of the comments alluded to. When working with massive data sets it gets pretty easy to start bumping your head on the 2GB file size limit in Access.
"Fast" is another matter. I've largely stopped working with Access because of, amongst other things, what I consider to be unacceptably slow speed when working with large data volumes. Wherever the name "JET" came from[1], it sure as Hades wasn't a metaphor for its speed.
The one advantage that Access does have is that it will allow you to pretty easily (if not quickly) produce Excel-style analysis reports (including some graphical reports) and also allow you to use VBA in the creation of them. SQL Server on the other hand, which is my data cruncher du jour, is much faster but barely has any front end at all. Probably the ideal situation would be (as also alluded to) to use either Access or Excel to do the front end and SQL Server to do the heavy lifting UNLESS (better still) you can do that lifting in Oracle rather than saving the data locally, then just have Excel/VBA create your reports on the summarised queries that you pull directly from Oracle.
I'd shy away from using Excel if the reports run to millions of lines but realistically, no report that runs that long is going to be worthwhile anyway. It's a data dump, not a report.
[1] Yes, yes, I know, Joint Engine Technology.
I have some pretty complex reports to write. Some of them... I'm not sure how I could write an sql query for just one of the values, let alone stuff them in a single query.
Is it common to just pull a crap load of data and figure it all via code instead? Or should I try and find a way to make all the reports rely on sql?
I have a very rich domain model. In fact, parts of code can be expanded on to calculate exactly what they want. The actual logic is not all that difficult to write - and it's nicer to work my domain model than with SQL. With SQL, writing the business logic, refactoring it, testing it and putting it version control is a royal pain because it's separate from your actual code.
For example, one the statistics they want is the % of how much they improved, especially in relation to other people in the same class, the same school, and compared to other schools. This requires some pretty detailed analysis of how they performed in the past to their latest information, as well as doing a calculation for the groups you are comparing against as a whole. I can't even imagine what the sql query would even look like.
The thing is, this % improvement is not a column in the database - it involves a big calculation in of itself by analyzing all the live data in real-time. There is no way to cache this data in a column as doing this calculation for every row it's needed every time the student does something is CRAZY.
I'm a little afraid about pulling out hundreds upon hundreds of records to get these numbers though. I may have to pull out that many just to figure out 1 value for 1 user... and if they want a report for all the users on a single screen, it's going to basically take analyzing the entire database. And that's just 1 column of values of many columns that they want on the report!
Basically, the report they want is a massive performance hog no matter what method I choose to write it.
Anyway, I'd like to ask you what kind of solutions you've used to these kind of a problems.
Sometimes a report can be generated by a single query. Sometimes some procedural code has to be written. And sometimes, even though a single query CAN be used, it's much better/faster/clearer to write a bit of procedural code.
Case in point - another developer at work wrote a report that used a single query. That query was amazing - turned a table sideways, did some amazing summation stuff - and may well have piped the output through hyperspace - truly a work of art. I couldn't have even conceived of doing something like that and learned a lot just from readying through it. It's only problem was that it took 45 minutes to run and brought the system to its knees in the process. I loved that query...but in the end...I admit it - I killed it. ((sob!)) I dismembered it with a chainsaw while humming "Highway To Hell"! I...I wrote a little procedural code to cover my tracks and...nobody noticed. I'd like to say I was sorry, but...in the end the job ran in 30 seconds. Oh, sure, it's easy enough to say "But performance matters, y'know"...but...I loved that query... ((sniffle...)) Anybody seen my chainsaw..? >;->
The point of the above is "Make Things As Simple As You Can, But No Simpler". If you find yourself with a query that covers three pages (I loved that query, but...) maybe it's trying to tell you something. A much simpler query and some procedural code may take up about the same space, page-wise, but could possibly be much easier to understand and maintain.
Share and enjoy.
Sounds like a challenging task you have ahead of you. I don't know all the details, but I think I would go at it from several directions:
Prioritize: You should try to negotiate with the "customer" and prioritize functionality. Chances are not everything is equally useful for them.
Manage expectations: If they have unrealistic expectations then tell them so in a nice way.
IMHO SQL is good in many respects, but it's not a brilliant programming language. So I'd rather just do calculations in the application rather than in the database.
I think I'd go for some delay in the system .. perhaps by caching calculated results for some minutes before recalculating. This is with a mind towards performance.
The short answer: for analysing large quantities of data, a SQL database is probably the best tool around.
However, that does not mean you should analyse this straight off your production database. I suggest you look into Datawarehousing.
For a one-off report, I'll write the code to produce it in whatever I can best reason about it in.
For a report that'll be generated more than once, I'll check on who is going to be producing it the next time. I'll still write the code in whatever I can best reason about it in, but I might add something to make it more attractive to use to that other person.
People usually use a third party report writing system rather than writing SQL. As an application developer, if you're spending a lot of time writing complex reports, I would severely question your manager's actions in NOT buying an off-the-shelf solution and letting less-skilled people build their own reports using some GUI.