ODI is mandatory at work but any changes made to SQL
are stored internally
I wish to programmatically pull out all SQL from
ODI store it in source control and then I can diff
and at least use half decent code reviews (worrying about who changed what
is far less of a problem than wondering if people are remembering
all their changes
So - how can I programmatically pull out Sql from ODI
Please visit www.odietamo.org.uk
Thats how we solved it
Related
I have tried to read up on this topic and I am still a bit unclear how to proceed. This seemed like a fairly basic task but it has been nowhere as simple as I had assumed. I have several SQL queries written and I want to be able to schedule them to run on a certain day each month and then automatically be exported to a .csv file in a selected folder. This will then allow them to be automatically uploaded into a BI and reporting tool that our firm uses (this part I know how to take care of).
I am fairly well versed in the writing of SQL queries, but everything beyond that I am pretty lost on. Right now I am using Microsoft SQL Management Studio 17. I thought that maybe scheduling jobs using the SQL Server Agent would be the solution, but the more I read about that and go down that path, the less I am convinced that it will allow me to export the query results into the .csv file that I need for it to be picked up. It is also important that these results are exported without headers.
Does anyone have any solutions for this? I am happy to answer any follow up questions if I am at all unclear.
You can create a job within the SQL server management studio to handle the whole thing.
http://social.msdn.microsoft.com/Forums/en/transactsql/thread/7d2280cf-3b33-46f7-ba82-4131e8a841c0
I have repository connected with the integrated source control in oracle. I did full export without any data from tables of my database as separate directories and it looks fine. The problem is that every time I want to change something I have export the stored procedure or table sql code and then to upload it in the repository, which is hard because at the end of the day I'm not sure about how many changes I did and I can forgot some of them. Full export without data could have been the solution but I don't have the time to wait for export 20-25 minutes at the end of every day. Is there any way to just export the changes which are made at the current day or made after the last export. Or maybe directly export the sql code on each compilation inside oracle management studio? The database is not on my computer its located in a server which I'm connected to.
here is how my git folder looks like in separate folders
You need to work the other way around. To change a package for example, open the corresponding source code file from your Git repository in your development tool (SQL Developer, PL/SQL Developer etc), make your changes, test, save the file, check in with ticket number and comment. As a rule you should not edit stored code directly in the database. (PL/SQL Developer has a checkbox "Allow editing of database source", which I generally leave unchecked. Probably other tools have something similar.)
I created a Tabular model using VS 2013 (because later versions don't support BIDS helper (which we use for creating a folder structure organizing all measures and dimensions within the model)). After working on it for a while I noticed that I was missing a table, so I went back to go get it. The only mistake which I made was that I ended up creating a new data connection in the process. Short of dropping those tables and recreating them, is there a simple way for me to change the data source of that new table so that I can delete the redundant connection to my database?
A colleague shared the following link: http://jakubka.blogspot.com/2013/11/how-to-change-connection-string-for.html.
While changing the code in this way most certainly comes across as being hacky and is probably not the highly recommended way to do things; it worked! Took me less than 5 minutes to fix and it was as simple as described in the link...
Jakub K (author of the article), thanks, man!
I encountered this issue today. I have VS 2017 and co-worker showed me how to do this.
Create New Data source in the Model(.bim file)
Save and close .bim file
Right click the .bim file and view code . This gives you an editable view.
Edit the connection on all the tables you want changed, not the original connection.
Save and exist .bim file
Open .bim file and confirm change.
In this case I use Tabular Editor.
There is a good article about it Tabular Editor Tricks - Convert to Legacy
I'm a programmer (mostly C++) who has moved into a non-software workplace. However, I don't have much experience with database stuff at all.
TL;DR: If we compare Crystal Reports to just writing scripts that execute SQL queries and parse the results, is there anything that CR can do that isn't possible via SQL queries & scripts? I'm talking purely in terms of extracting data - not making pretty documents.
Detail:
At my workplace they have a process where you run a bunch of Crystal Reports, modify the date range to the current month, manually export each to excel, delete the rows and columns that aren't needed, and then cut and paste into a summary excel document that is used by management.
To me, this is pretty crazy and stupid. I'd like to automate/script most of it.
So I have two options:
Learn Crystal Reports and try to modify the existing reports to be more automated.
Dump CR and just learn SQL and do the whole thing programmatically with scripts working with CSV files or something.
I'd much rather learn SQL since it's more general and useful. But I need to be assured that I can get the data output that I need (without writing a million lines of code to reproduce CR myself.)
So yeah, I'm looking for an answer like, "The two are equivalent. Anything you can do in CR you can do easily via scripts and SQL," or "If you need to group records into categories based on a parameter and then sum their one of their fields, then CR will do it much more easily than raw code," to push me in one direction or another.
Edit:
Some additional detail. At the moment my crystal reports run a database query, and then crystal does things like, "don't display the records that are returned, instead group the records by Field A and then display the count of how many records in each group."
Is functionality like this difficult to reproduce via SQL coding? I wouldnt want to have to write a python (or whatever) script to parse and manipulate the data from plaintext CSV, for example.
You can't just compare SQL and CR - they have different purpose. SQL (in this context) is data source, CR is pretty output formatter. For excel you would need data, not formatted output. Excel combined with SQL can give you all CR options (dynamic crosstab reports, charts etc) what you can't get directly from SQL data.
BTW, creating SQL views or procedures is often needed to overcome CR limitations; from this standpoint SQL has lot of more options than CR.
I personally would go with SQL+Excel route. In our company we're using simply SQL+CR without postprocessing, sometimes SQL+Excel. Our customers are using different approaches.
But like said by other people, choice of tools depends on more things. Who has to redesign reports? Who will maintain these reports? How often requirements change? Are there more uses for CR reports besides sourcing Excel tables? Who will be waked up at night, if reports do not work?
Management perpective:
In many I will say mostly cases management does not know SQL. So if a manager for E.g.HR wants to know staus about something then how he will get that status?? This is where Crystal reports come into picture, Using crystal reports they do not have to worry about SQL; they will just enter required fields and get their data.
Programmer perspective:
Simple data outputs can be achieved through SQL but consider a scenario where you need to pull details as well as summary. I agree it can be done via SQL but consider the overhead of time and proficiency required to develop such output using sql. I bet it wont be that easy to develop such output using sql as compared to crystal. So I will say learn both SQL and crystal, you will get to choose the tool to apply for your requirement.
You can write SQL and drop it into the Crystal Report. Best of both worlds, and possibly faster performance than the drag-and-drop Crystal functionality.
You will see some response time lag when the report runs.
There are actually a few things that Crystal Reports can do that are very tricky using plain SQL Queries as Crystal Reports can access the entire dataset in a single formula and can do things at runtime.
However unless you have some really crazy complex Crystal Reports I would recommend building a tool in Excel that can one click the info straight into a new sheet.
I did this and it got me a promotion, not kidding :P
I have a custom Excel Addin I can give you code to that basically does this:
On open, connects to the database and downloads a list of menu options connected to views and procedures
Adds these menu options into a new Ribbon tab within Excel
When one is clicked, runs the view and dumps the entire dataset (properly formatted) into a new sheet
Advantage of this is you can update the main menu list and each view it references without making any changes to the file or re-issuing anything to everyone.
Crystal could be helpful if you want to create a document with a specific layout , logos etc. and show some data on it. Export to excel from Crystal repot is not easy - usually there are a lot of empty columns and rows and each report should be tweaked to avoid that.
If you need to export some data from a SQLServer database to excel your best option will be SSIS ( I guess you have a license for SQL Server). If you don't have license for SSIS or you are using for example Access database there are also some inexpensive tools, which can retrieve data from any database ( not just SQLServer) and export it to excel. I would suggest you to check this one: http://www.r-tag.com. It can run Crystal reports and SQL reports so you can start using your crystal reports immediately and start transforming them to SQL reports whenever you have time for that. Both reports could be exported to excel.
i fixed this by editing excel sql, Left(Column_maxLength, 250)
this resolved my issue
in my case if even if i read left 250 character is enough
I was wondering if there is a way to automatically append to a script file all the changes I am making to my columns, tables, relationships etc...
The thing is I am doing a lot of different changes on a TEST db and the idea will be to apply this change script when I move the test db to production... hence keeping production data but applying all schema and object changes.
Is there an easy way to do this? Can it also migrate database diagram changes?
I have seen how you can create a change script each time I do a change but this means I have to copy and paste into a master file. Actually pretty easy!
I was just wondering if I was missing something?
Do not make changes to the test server using the UI. Write scripts and keep them under source control. You can test your scripts starting from backups of the live data and you can tune yoru scripts untill they achieve the desired result. Then you can check in the scripts for reference and later apply them on the live server. See this article Version Control and Your Database.
BTW, check out the SSMS toolpack, I think it may do what you want (I'm not sure). My advice stand none the less: version your schema, use explicitly created/saved scripts, use source control.
There's no way to directly generate a "delta" script in SSMS.
However, if every time you publish changes, you script out the entire database, including data, to SQL using the SQL Server Database Publishing Wizard you should be able to extract diffs between the versions and get your deltas that way.
If money is no object, you can purchase Visual Studio Team System Database Architect edition and use its fantastic database comparison tools to generate and version control exactly the diffs you want.
Try using TableDiff , that came with SQL Server 2005.
SQL Server 2005 TableDiff Utility
tablediff Utility
We have the process where when a developer gets done with a change, they then script it out and check it into Subversion. In Subversion we have a folder for Tables, Stored Procs, Data, etc. They script it out so it is repeatable (i.e. don’t insert the new data if it is already there.) This is important to do anyway so you keep the history of changes for a given object in the database.
In the past, we would just enter each of the files that we wanted scripted out into a text file (i.e. FileListV102.txt). When we were ready to make a release we would do “get latest” on all of the files (from VSS back then.) We then had a simple utility that would read the “file list” file and open each of those files in turn concatenating them into an output file. That is pretty easy to code.
We outgrew that and now we have a release management tools (which can be found here and will be on sale mid September), that takes all of the files and creates a big SQL script file out of it. It does it in the order that you would expect based on the folder names – so files found in the "Tables" folder are done before those in the "Data" folder, etc.
Either way, once you are done you have a big SQL script file that you can then apply to a fresh copy of production and that is what you test against.
I know I'm way late to the party, but I just wanted to add that there are tens of third party products out there. Some are very good, some are very cheap or free, and some are a mixture. I listed 22 here:
http://bertrandaaron.wordpress.com/2012/04/20/re-blog-the-cost-of-reinventing-the-wheel/
We have been using a relatively new software called Kal Admin.
It has Change Management feature and let distributing selected changes to other databases very easily. We used to do it by comparing two databases but it not satisfy our need for change tracking.
BTW Kal Admin has Metadata and data compare capabilities as well.