Broadly speaking, can someone tell me if I'm headed in the right direction?
I now know how to write SQL Queries pretty well.
I would like to start aggregating multiple queries onto one "form"/template (not sure if that's the correct terminology).
I have access to lots of clean data in the form of Excel Files.
I should be able to load the excel files into Visual Studio and then write reports that refer to those excel files as databases, am I right?
I haven't been able to find a great SSDT tutorial yet, but I'll keep looking. Thanks for any feedback.
First off, I apologize that I'm writing a bit of a novel here. My understanding of your question is that you're looking for architectural guidance on the best way to go, and that's not a quick answer.
If, however, I've misinterpreted your intent and you are actually just looking for how to code up an excel file as a database, there is already a lot of articles online that you can google.
Regarding your architectural question...it is really going to come down to choosing the best trade-offs for what you're building. I will give you some pointers that I have learned and hopefully it is helpful to you and others in the community.
I would be very hard pressed to advise that you use an excel file as a database.
While it might seem like the most straight forward solution, the trade-offs here are very costly in debugging file locking issues and dealing with excel specific errors, it becomes a death by a thousand cuts. It is certainly possible, but this is a trap that I personally fell into early in my career.
Here's is a link to some descriptions of the problems that you'd have with an excel file database and here is a 2nd link.
To paraphrase your question, it sounds like you're developing a personal ETL application for improving your productivity and your company's metrics. Spreadsheets come into your e-mail inbox and transformed versions of the spreadsheets go out of your e-mail outbox. You are wanting to look at the departments' data from a historical and comparative perspective. I have done this many times in the past as well and it is a very reasonable goal.
The best way that I have found to do this is to use a SQL Server database. You can start this out in phases of minimal viable product to do this in small easy chunks.
Phase I:
Download and install SQL Server 2016 Express free. Make sure
to install localdb when you install SQL Server 2016. See the localdb
instructions for more information.
Create the localdb instance on the command line.
Connect to the new localdb instance in SQL Server
Create a new Database that you'll use for importing the data. Give it a name like "ReportData"
Import the excel files received from the variety of businesses into the new database. This stackoverflow answer gives a great description of how to do it. Here is an alternate example.
If you get any error messages about drivers you may need to download the correct drivers.
Develop your SQL queries that you want to use. For simplicity, I'm just showing a basic select statement here, but you can build some sophisticated SQL queries for aggregating the data in this step.
Export the data from the excel file into a CSV file or an excel file. You do this by right clicking in the "Results" area and selecting "Save Results As..."
Manually copy and paste the resulting values into the excel templates that you would like.
Note step 9 will be automated soon, but it is better for now to understand your domain objects and be thinking about the business logic that you're building in a quick iterative manner.
Phase II:
Create a new Console application in Visual Studio that will transform the data from the database into an Excel file output. The most powerful way to do this is to use EPPlus. Here is a detailed explanation on how to do this.
Note, when you run the source code from the detailed explanation link, you need to change the output path first, for example to c:\temp. Note also that there are plenty of other Excel spreadsheet helper packages out there, so feel free to look around at other packages as well. EPPlus is simply one that I have been successful with in my projects. This Console application will be querying your SQL Server database using the queries that you built in step 7 above.
Phase III:
In time, you many find that co-workers and managers within your company want to start accessing your data directly through a web page...
At a high level, the steps you would take are:
Backup the database and restore it onto a server.
Implement a simple MVC application
Perhaps even build web pages to allow users to import excel so that they don't need to e-mail them to you any longer.
An additional note, there are Enterprise level ETL and reporting tools out there as well, such as SSIS/SSRS, etc that you could look into if you're looking for a more sophisticated tool set, but I didn't get that impression with your question.
I hope that this answer helps and isn't too long winded. Please let me know if any of the steps are unclear, I know it's a lot of information in one post.
Related
I am working using Tableau and have to write down multiple different SQL each time, while making new data sources.
I have to save all changes on SQL for every data source.
Currently I would paste the SQL on notepad and save them on separate folder in my computer, along with description of the changes.
Is there any better way to do this?
Assuming you have permission to create objects in the database, begin by creating database views, As #Nick.McDermaid commented.
Then, instead of using Custom SQL data source in Tableau, just connect to the View as if it were a table.
If you need to track the changes to these SQL views of your data, you will need to learn how to use source control for the .sql files that can be scripted from within SQL Server Management Studio:
Your company or school may have a preferred source control system already in use, in which case you should use that. If they don't, or if you are learning at home, then Git and Subversion are popular open source choices.
There are many courses available on learning platforms like Coursera that will teach you how to learn how to use those systems.
I had similar problem as you.
We ended up writing the queries in SQL Editor SQL Work bench (https://www.sql-workbench.eu/), then managed the code history and performed code peer-review (logic, error check, etc) in team shared space (like confluence).
The reasons we did that is
1) SQL queries are much easy to write on Work Bench
2) Code review is a must! You will find through implementing a review process more mistakes than you could ever think about
3) The shared space is just really convenient as it is accessible by everyone, and all errors are documented. After sometimes you get a lot of visible knowledge accumulated.
I also totally agree with Nick as this is one step to a reporting solution. But developing a whole reporting server is heavy, costly and takes time. Unless management are really convinced of the importance of developing a reporting solution, you may have to get a workaround with queries and Tableau (at least that was the case for us)
A little late to the party, but I would suggest you simply version the tableau workbook. The contents of the workbook are XML, so perfect for versioning using file based tools (Dropbox, One Drive, etc.) or source control (git, etc.). The workbooks themselves are usually quite small, so just make sure to keep the extract data separate if you use it.
Just a bit of background on where my question is coming from: my company has multiple databases across the globe that uses the same schema and once of my department's responsibility is to monitor and make sure all these DBs are in sync from a schema SQL change perspective.
Now, my question is if anyone knows of any Software/tool that has a a Frontend UI which is able to do the following (the lower number the more important to have):
Able to track what SQL code change was applied on which database and when. Basically, if we write a SQL query that changed the structure of a table and we need it applied to 80% or 100% percent of the DBs, either via manual input or some automatic check the tool will tell me that yes, this was indeed applied.
Code distribution tool: we give it the query or a file that contains the code and it's able to push to the Databases it needs to (and create the audit log for that)
Code/object repository: keeps track of what was custom developed and pushed to the databases
I know SSIS might be able to do some of these things, but we need a tool that also has a simple frontend interface that can be accessed by non-IT personnel. (*clarification: we are not planning on giving non-DBA people access to change things, just to the audit aspect of said tool)
I've tried searching the internet, but i have a feeling i'm not using the right vocabulary to get the results i'm looking for.
Hence i wanted to see if the community was aware of any such tool or something similar?
Try searching for one of these two types of systems:
Release/Build/Deployment Automation Complex programs like Serena that have modules for pushing, tracking, and auditing any kind of software, anywhere. These will include all the GUI bells and whistles. But you'll have to deal with extra databases, configuration, agents, workflows, consultants(?), etc. These programs are geared more towards developers.
Remote Execution/Configuration Management Simpler programs like Salt, Fabric, and Ansible that let you run operating system commands anywhere. They don't offer as many features, and you have to do more of the work yourself, but in some ways that's liberating. If you know exactly what commands you want to run you don't need some other program holding your hand. These programs are geared more towards administrators.
From a database administrator's point of view, the main problem with those types of programs is that none of them are relational. Yes they can connect to a database and run a script, but none of them really speak SQL. Their native languages are Java, XML, SSH, etc. There's nothing wrong with those technologies, but if you only care about databases you don't want to deal with all that complexity.
If you're not happy with either of those types of programs I recommend you look at my open source program Method5. It is a remote execution program built as an extension to Oracle SQL. It works entirely inside an Oracle database, so you can install it yourself and won't need any additional websites, agents, configuration files, GUIs, etc.
Based on your comment about getting bogged down by links, and my answer to your question about half a year ago, I think this is the kind of program you were gradually heading towards creating. It took my team a couple thousand hours of developing and testing to get it right so you were probably wise to give up on making your own.
To specifically answer your requirements:
Tracking Changes are stored in an audit trail. But more importantly it has the ability and a pre-built script to compare an unlimited number of schemas, all in one view. At the end of the day what you really want to know is "are my schemas the same", not necessarily "did the same thing get run everywhere?".
Code Distribution If you just have SQL or PL/SQL, deploying it through Method5 is as easy as it can possibly get. Just specify what you want to run, and where you want to run it, like this: select * from table(m5('create index ...', 'dev, qa, prodDB1, prodDB2')); The program does not (yet) run SQL*Plus scripts. But when you have the ability to run SQL and PL/SQL so easily there's little need for SQL*Plus.
Code Repository All executions are stored in a simple table, M5_AUDIT. It contains the code, who ran it, where they ran it, and how they ran it. It wasn't designed to be a repository like SVN but it's good enough for simple auditing and tracking code.
Method5 does not contain a GUI but in some ways I consider that to be a feature. Since everything is done relationally, everything is in a simple table. You can use any of your existing GUIs - Toad, PL/SQL Developer, Excel, Apex, etc. It's a robust back-end solution that will hopefully make a good foundation for easily building a simple front end.
I have a database that had 15 years of cruft stuffed into it by multiple teams and people in multiple languages. I am looking to rename tables/columns/constraints to match some type of a standard.
The problem is that a column may be referenced in a ton of stored procs and there is no way to find other than search each sproc with a tool like SQL Digger. The problem is that I want to rename a massive amount of entities and doing it manually for each sounds painful.
I've been looking for a tool that helps in name refactoring and can't find anything. Some tools [here] vaguely claim to do that, but don't really (I haven't looked at all the ones listed to be fair).
Has anyone had experience with such a tool?
I'm using the ApexSQL Refactor for some time. It is a freeware tool and so far it works very well.
There's an article "How to change an object name without breaking your SQL database" in their solution center.
I've noticed a new version announcement (2013), however I am not sure if it will remain free or not.
Be careful when updating objects using any of the tools mentioned above because they only search for references in you database. Important thing to keep in mind is the code you have in data access layers of your applications.
Another tool you might want to check out is ApexSQL Clean. It can find all unused database objects, show all references visually but it also searches for .NET solutions and finds references there.
Again, considering that your database is 15 years old you probably have code in different legacy applications and not only .NET. Anyway, good luck. Hope this is all done by now :)
I have used database projects (in Visual Studio 2010) for refactoring activities in the past with a good deal of success. Database projects definitely have a number of quirks but nothing you can't work around.
You can find more details about it here: http://msdn.microsoft.com/en-us/library/dd193420
I have not used this, but have used other of their Sql Server tools, and they did what they claimed.
http://www.red-gate.com/products/sql-development/sql-prompt/
A couple of options:
SQL Server Data Tools (review here)
Red-Gate SQL Refactor (well, now part of SQL Prompt)
Note that these tools can only be smart enough to find references that are exposed through direct reference or through proper dependencies. If you construct a table or column name using dynamic SQL, you're out of luck.
I also blogged about keeping sysdepends up to date a few years ago, however I'm not sure how useful it will be with columns in particular:
https://sqlblog.org/2008/09/09/keeping-sysdepends-up-to-date-in-sql-server-2008
I am primarily a web developer, mostly working with PHP, MySQL, and JavaScript. I was recently contacted by a local Sheriff's Office (small town word of mouth, nerds are always needed) to digitize a 4 page monstrosity of a form... because nobody could read the handwriting of the deputies.
The catch here is that this is a small town department and, while they are fancy enough to carry computers in the field, they are not connected to the Internet. Visual Basic was the first solution that came to mind and I have been scrambling to learn the basics. I am confident in my ability to organize the content of the form and perform any necessary validation but I am unsure where to begin in terms of storing each report locally (database) and printing the end result.
Another matter that makes things complicated is the fact that they want the end result to look exactly the same as the original form, only typed instead of hand written.
So, to sum things up, here are the questions I have:
There seem to be several options for databases in VB 2010 Express. What is the best option for LOCAL storage of records?
It looks as though the best way to format the form the exact way they want it to look with populated data would be to create a form within the application with just this content on it. Is this the best solution or might there be a better way - possibly outputting to another file? And if the data is put on another form, how would I go about printing it?
Many thanks!
The word "best" is of course subjective, so instead I'll give you some pros and cons for a database.
SQL Server Express is a really awesome database to work with that acts almost exactly like the big paid version. Some high-level things like replication and encryption aren't supported but you don't have a need for that probably. I've built many websites that target it with zero performance problems. The downside of SQL Server Express is that you need to install it on every machine and it pretty much needs to be running all the time. It doesn't "weigh" a whole lot but its still going to be running in the background 24/7. If you create an installer from within Visual Studio/VB Express (which you should) you can check it as a prerequisite and the installer will pretty much take care of it for you. As a major security target you are opening a potential for security issues which you should be aware of.
SQLite would another great choice, there's some great .Net wrappers available. If you're used to using SQL Server or MySql you might find SQLite limiting but you get used to it. SQLite doesn't have a "database engine" and its goal is to be a very lightweight open source SQL database system.
The third option that I'd recommend is just writing to an XML file. Simple, no engine, no tables, no third-party whatever, just raw text that anyone can parse if something breaks. EDIT And VB.Net has some wonderful built-in XML syntactic things such as XML literals:
Dim MyXml = <Person>
<FirstName><%= txtFirstName.Text %></FirstName>
<LastName><%= txtLastName.Text %></LastName>
</Person>
For the form generation, I'd recommend using something like iTextSharp. (Free but make sure you check that the license matches yours.) Take their actual PDF Form (or create a PDF of theirs), use Acrobat or something similar to turn it into a "PDF Form" and then just use iTextSharp to fill in the form. There's a bunch of support on this site if you've got any questions about it.
Is there a tool or programming method available to take the layout defined in a PDF and import it into Sql Server Reporting Services 2005?
Quite a few layouts are created as PDFs by third parties. It would make development a lot easier if I could use some of the layout already defined.
I do not know of anything that will do this.
The only way I could think of is to convert the PDF file into a RDLC file. The RDLC file is a XML file, and follows a specific schema. To do this, would take a lot of time. Your probably better of creating the reports in report services manually.
I will definitely keep an eye out on this thread, however, I am quite certain that this solution does not yet exist.
Jon's answer concerning turning the PDF into XML and using that as a base sounds like a possible solution, but as he said the time and effort behind it probably wouldn't make it worth it at all.
As far as I know, I don't recall any sort of true layout-importing feature to SSRS, nor have I heard of any third party tools which would make it easier. If I had, I would have already purchased it, because a major sink of my own time is replicating forms and reports generated on a client's legacy system to work on SSRS.
I currently have the same problem, but if I have to, I will take the pdf and save them as images -- allowing me to insert them into the report -- as a last resort.