I have a large collection of SAS programs. Currently, I have lots of SAS modules licensed and installed (I've checked which ones via %sasinstallreporter), but I'm not sure whether I need all of them.
How can I determine which modules I actually need in order to run all of my code? They're rather expensive and I don't want to pay for modules that I'm not going to use. Is there a list of procs / libname engines somewhere that are tied to specific SAS modules?
Apologies if this is a duplicate, but some brief searches didn't turn up anything obvious.
I have posted a follow-up question here regarding what other things I should be looking for in my code.
1st, get a list of all the procedures your code uses. You can do this manually, or write a script in SAS/AWK/Perl to parse through your code.
Next, go to http://support.sas.com/documentation/ and search each for each of the unique Procedures.
For example, if I search for PROC REG, I get these results (http://support.sas.com/cdlsearch?charset=iso-8859-1&nh=25&ct=80000&qt=PROC+REG++&s1=3&searchterm=PROC+REG&sasreleasepart=&sasprodpart=&searchContains=). You will see under the results SAS/STAT(R), which tells you it is part of the SAS/STAT module.
You may also have things licensed that do not show up as PROC XXX in your code. Things like SAS/Integration Technologies are used for running and connecting to a SAS server. Talk to your SAS administrator about how those are used.
Happy hunting.
If you are paying for SAS, I would say send this question to SAS technical support, they should be able to help you with a list.
I don't think there is a definative list of modules and associated PROCS on the web.
Regards,
Vasilij
Related
I am working using Tableau and have to write down multiple different SQL each time, while making new data sources.
I have to save all changes on SQL for every data source.
Currently I would paste the SQL on notepad and save them on separate folder in my computer, along with description of the changes.
Is there any better way to do this?
Assuming you have permission to create objects in the database, begin by creating database views, As #Nick.McDermaid commented.
Then, instead of using Custom SQL data source in Tableau, just connect to the View as if it were a table.
If you need to track the changes to these SQL views of your data, you will need to learn how to use source control for the .sql files that can be scripted from within SQL Server Management Studio:
Your company or school may have a preferred source control system already in use, in which case you should use that. If they don't, or if you are learning at home, then Git and Subversion are popular open source choices.
There are many courses available on learning platforms like Coursera that will teach you how to learn how to use those systems.
I had similar problem as you.
We ended up writing the queries in SQL Editor SQL Work bench (https://www.sql-workbench.eu/), then managed the code history and performed code peer-review (logic, error check, etc) in team shared space (like confluence).
The reasons we did that is
1) SQL queries are much easy to write on Work Bench
2) Code review is a must! You will find through implementing a review process more mistakes than you could ever think about
3) The shared space is just really convenient as it is accessible by everyone, and all errors are documented. After sometimes you get a lot of visible knowledge accumulated.
I also totally agree with Nick as this is one step to a reporting solution. But developing a whole reporting server is heavy, costly and takes time. Unless management are really convinced of the importance of developing a reporting solution, you may have to get a workaround with queries and Tableau (at least that was the case for us)
A little late to the party, but I would suggest you simply version the tableau workbook. The contents of the workbook are XML, so perfect for versioning using file based tools (Dropbox, One Drive, etc.) or source control (git, etc.). The workbooks themselves are usually quite small, so just make sure to keep the extract data separate if you use it.
Broadly speaking, can someone tell me if I'm headed in the right direction?
I now know how to write SQL Queries pretty well.
I would like to start aggregating multiple queries onto one "form"/template (not sure if that's the correct terminology).
I have access to lots of clean data in the form of Excel Files.
I should be able to load the excel files into Visual Studio and then write reports that refer to those excel files as databases, am I right?
I haven't been able to find a great SSDT tutorial yet, but I'll keep looking. Thanks for any feedback.
First off, I apologize that I'm writing a bit of a novel here. My understanding of your question is that you're looking for architectural guidance on the best way to go, and that's not a quick answer.
If, however, I've misinterpreted your intent and you are actually just looking for how to code up an excel file as a database, there is already a lot of articles online that you can google.
Regarding your architectural question...it is really going to come down to choosing the best trade-offs for what you're building. I will give you some pointers that I have learned and hopefully it is helpful to you and others in the community.
I would be very hard pressed to advise that you use an excel file as a database.
While it might seem like the most straight forward solution, the trade-offs here are very costly in debugging file locking issues and dealing with excel specific errors, it becomes a death by a thousand cuts. It is certainly possible, but this is a trap that I personally fell into early in my career.
Here's is a link to some descriptions of the problems that you'd have with an excel file database and here is a 2nd link.
To paraphrase your question, it sounds like you're developing a personal ETL application for improving your productivity and your company's metrics. Spreadsheets come into your e-mail inbox and transformed versions of the spreadsheets go out of your e-mail outbox. You are wanting to look at the departments' data from a historical and comparative perspective. I have done this many times in the past as well and it is a very reasonable goal.
The best way that I have found to do this is to use a SQL Server database. You can start this out in phases of minimal viable product to do this in small easy chunks.
Phase I:
Download and install SQL Server 2016 Express free. Make sure
to install localdb when you install SQL Server 2016. See the localdb
instructions for more information.
Create the localdb instance on the command line.
Connect to the new localdb instance in SQL Server
Create a new Database that you'll use for importing the data. Give it a name like "ReportData"
Import the excel files received from the variety of businesses into the new database. This stackoverflow answer gives a great description of how to do it. Here is an alternate example.
If you get any error messages about drivers you may need to download the correct drivers.
Develop your SQL queries that you want to use. For simplicity, I'm just showing a basic select statement here, but you can build some sophisticated SQL queries for aggregating the data in this step.
Export the data from the excel file into a CSV file or an excel file. You do this by right clicking in the "Results" area and selecting "Save Results As..."
Manually copy and paste the resulting values into the excel templates that you would like.
Note step 9 will be automated soon, but it is better for now to understand your domain objects and be thinking about the business logic that you're building in a quick iterative manner.
Phase II:
Create a new Console application in Visual Studio that will transform the data from the database into an Excel file output. The most powerful way to do this is to use EPPlus. Here is a detailed explanation on how to do this.
Note, when you run the source code from the detailed explanation link, you need to change the output path first, for example to c:\temp. Note also that there are plenty of other Excel spreadsheet helper packages out there, so feel free to look around at other packages as well. EPPlus is simply one that I have been successful with in my projects. This Console application will be querying your SQL Server database using the queries that you built in step 7 above.
Phase III:
In time, you many find that co-workers and managers within your company want to start accessing your data directly through a web page...
At a high level, the steps you would take are:
Backup the database and restore it onto a server.
Implement a simple MVC application
Perhaps even build web pages to allow users to import excel so that they don't need to e-mail them to you any longer.
An additional note, there are Enterprise level ETL and reporting tools out there as well, such as SSIS/SSRS, etc that you could look into if you're looking for a more sophisticated tool set, but I didn't get that impression with your question.
I hope that this answer helps and isn't too long winded. Please let me know if any of the steps are unclear, I know it's a lot of information in one post.
Just a bit of background on where my question is coming from: my company has multiple databases across the globe that uses the same schema and once of my department's responsibility is to monitor and make sure all these DBs are in sync from a schema SQL change perspective.
Now, my question is if anyone knows of any Software/tool that has a a Frontend UI which is able to do the following (the lower number the more important to have):
Able to track what SQL code change was applied on which database and when. Basically, if we write a SQL query that changed the structure of a table and we need it applied to 80% or 100% percent of the DBs, either via manual input or some automatic check the tool will tell me that yes, this was indeed applied.
Code distribution tool: we give it the query or a file that contains the code and it's able to push to the Databases it needs to (and create the audit log for that)
Code/object repository: keeps track of what was custom developed and pushed to the databases
I know SSIS might be able to do some of these things, but we need a tool that also has a simple frontend interface that can be accessed by non-IT personnel. (*clarification: we are not planning on giving non-DBA people access to change things, just to the audit aspect of said tool)
I've tried searching the internet, but i have a feeling i'm not using the right vocabulary to get the results i'm looking for.
Hence i wanted to see if the community was aware of any such tool or something similar?
Try searching for one of these two types of systems:
Release/Build/Deployment Automation Complex programs like Serena that have modules for pushing, tracking, and auditing any kind of software, anywhere. These will include all the GUI bells and whistles. But you'll have to deal with extra databases, configuration, agents, workflows, consultants(?), etc. These programs are geared more towards developers.
Remote Execution/Configuration Management Simpler programs like Salt, Fabric, and Ansible that let you run operating system commands anywhere. They don't offer as many features, and you have to do more of the work yourself, but in some ways that's liberating. If you know exactly what commands you want to run you don't need some other program holding your hand. These programs are geared more towards administrators.
From a database administrator's point of view, the main problem with those types of programs is that none of them are relational. Yes they can connect to a database and run a script, but none of them really speak SQL. Their native languages are Java, XML, SSH, etc. There's nothing wrong with those technologies, but if you only care about databases you don't want to deal with all that complexity.
If you're not happy with either of those types of programs I recommend you look at my open source program Method5. It is a remote execution program built as an extension to Oracle SQL. It works entirely inside an Oracle database, so you can install it yourself and won't need any additional websites, agents, configuration files, GUIs, etc.
Based on your comment about getting bogged down by links, and my answer to your question about half a year ago, I think this is the kind of program you were gradually heading towards creating. It took my team a couple thousand hours of developing and testing to get it right so you were probably wise to give up on making your own.
To specifically answer your requirements:
Tracking Changes are stored in an audit trail. But more importantly it has the ability and a pre-built script to compare an unlimited number of schemas, all in one view. At the end of the day what you really want to know is "are my schemas the same", not necessarily "did the same thing get run everywhere?".
Code Distribution If you just have SQL or PL/SQL, deploying it through Method5 is as easy as it can possibly get. Just specify what you want to run, and where you want to run it, like this: select * from table(m5('create index ...', 'dev, qa, prodDB1, prodDB2')); The program does not (yet) run SQL*Plus scripts. But when you have the ability to run SQL and PL/SQL so easily there's little need for SQL*Plus.
Code Repository All executions are stored in a simple table, M5_AUDIT. It contains the code, who ran it, where they ran it, and how they ran it. It wasn't designed to be a repository like SVN but it's good enough for simple auditing and tracking code.
Method5 does not contain a GUI but in some ways I consider that to be a feature. Since everything is done relationally, everything is in a simple table. You can use any of your existing GUIs - Toad, PL/SQL Developer, Excel, Apex, etc. It's a robust back-end solution that will hopefully make a good foundation for easily building a simple front end.
I have a database that had 15 years of cruft stuffed into it by multiple teams and people in multiple languages. I am looking to rename tables/columns/constraints to match some type of a standard.
The problem is that a column may be referenced in a ton of stored procs and there is no way to find other than search each sproc with a tool like SQL Digger. The problem is that I want to rename a massive amount of entities and doing it manually for each sounds painful.
I've been looking for a tool that helps in name refactoring and can't find anything. Some tools [here] vaguely claim to do that, but don't really (I haven't looked at all the ones listed to be fair).
Has anyone had experience with such a tool?
I'm using the ApexSQL Refactor for some time. It is a freeware tool and so far it works very well.
There's an article "How to change an object name without breaking your SQL database" in their solution center.
I've noticed a new version announcement (2013), however I am not sure if it will remain free or not.
Be careful when updating objects using any of the tools mentioned above because they only search for references in you database. Important thing to keep in mind is the code you have in data access layers of your applications.
Another tool you might want to check out is ApexSQL Clean. It can find all unused database objects, show all references visually but it also searches for .NET solutions and finds references there.
Again, considering that your database is 15 years old you probably have code in different legacy applications and not only .NET. Anyway, good luck. Hope this is all done by now :)
I have used database projects (in Visual Studio 2010) for refactoring activities in the past with a good deal of success. Database projects definitely have a number of quirks but nothing you can't work around.
You can find more details about it here: http://msdn.microsoft.com/en-us/library/dd193420
I have not used this, but have used other of their Sql Server tools, and they did what they claimed.
http://www.red-gate.com/products/sql-development/sql-prompt/
A couple of options:
SQL Server Data Tools (review here)
Red-Gate SQL Refactor (well, now part of SQL Prompt)
Note that these tools can only be smart enough to find references that are exposed through direct reference or through proper dependencies. If you construct a table or column name using dynamic SQL, you're out of luck.
I also blogged about keeping sysdepends up to date a few years ago, however I'm not sure how useful it will be with columns in particular:
https://sqlblog.org/2008/09/09/keeping-sysdepends-up-to-date-in-sql-server-2008
We have a Delphi 2006 application that drives a MS SQL Server database.
We have found a vulnerability where it is possible to load the executable into a hex editor and modify the SQL.
Our long term plan is to move this SQL to CLR stored procedures but this is some way off since many of our clients still use SQL 2000.
We've thought about obfuscating the strings, does anyone have a recommendation for a tool for doing this?
Is there a better solution, maybe code signing?
Sorry for being blunt, but if you are thinking of applying "security" measures in your executable you are doomed. No scrambling schema will retain an average hacker.
You also haven't explained how is your app designed. Is the database hosted by you, or resides in your client's premises? If the latter, then just forget about security and start hiring a lawyer to get a good confidentiality contract so your clients behave. If the former, then using stored procedures is the easiest way.
If embedded SQL is being hacked, then it implies that your database is quite open and anyone with MSQRY32.EXE (that is, MS Office) can get your data.
If you are a vendor, then you can't rely on CLR being enabled at your clients. So, why not use non-CLR stored procedures and correct permissioning in the database that is version independent?
This is not a vulnerability. If your machines are vulnerable to having people locally modify EXEs, that is your vulnerability.
All EXEs can be hacked, if someone has local admin account access, your game is over long before they get near your resource strings.
It will never be possible to protect completely, but you can make "casual attack" harder. The simple system that I use is a "ROT47" type system which is like ROT13 but wider ranging. The code then gets to look like the following:
frmLogin.Caption := xIniFile.ReadString(Rot47('$JDE6>' {CODEME'System'}),
The key here is that I have a comment which includes the string so both I can see it, but more importantly so can the utility that I run in my FinalBuilder build script. This allows me to ensure that strings are up-to-date at all times in release code. The utility looks for {CODEME in the lines, and if found knows the format of the data to output appropriately.
A solution that would require a deep restructuring of the application would be to use a multi-tier approach - most the of the SQL code would be in the application server module, that being on a server should be more protected than a client side exe.
Can't you encrypt all your queries and put them to the resource file?
During runtime, firstly you would have to:
Load your query string from resource.
Decrypt it.
Then you just run your query as before.
That should not be a big problem. Of course if you are not storing your queries in some resource / folder than you need to refactor your application a bit. But you should store them anyway in some organized manner. So you will be hitting a two birds with one stone here ;-)
For encryption of the strings you could use a free library called DCPCrypt.
I think you should use a exe packer which makes it hard for anyone to modify the stuff using hex editor.
First - do an analysis of your threat. Who is using your vulnerability, why is this a problem. Then act accordingly.
If your application is win32 and your threat are some kids witch are just having fun, a free exe packer (e.g. upx) might be the solution. On .NET applications signing might be what you want.
If you need more than that, it's going to be expensive and it's going to be more difficult to develop your application. Perhaps you even need to restructure it. Commercial protection schemes are available (perhaps with dongle?) - even protection schemes where you store your strings on some external hardware. If the hardware is not present, no SQL-Strings. But, as I said, that's more expensive.
Move DB interface to stored procedures. Normal regular stored procedures without any CLR. It's not a big deal if you already have queries to put inside.
If you don't want to learn T-SQL for some reasons, simple move all you query string to database and store in application single query, which purpose is reading SQL code with given query ID from database only.
All tricks with encoding produces a lot of troubles, but don't give any real security because must use reversable encrypting (dictated by the nature of the problem) and all keys for decoding placed in application executable too.
There are "protection" suites that encrypt and/or validate your exe before running. searching for "encrypt exe" or "validate exe" or so will probably help. Usually they are payware, but sub $100.
The principle is the same as an exe packer (and has some of its downsides, like cheaper antivirus heuristics sometimes reacting on them, a slightly elevated memory load), just more focussed on security. A problem is also that for most exe packers, depackers exist.
I use dinkeydongle's wares, but that is a kind that also ties into an hardware dongle, so that might be a bridge to far for you.