This question pertains to the i5/OS (mainframe, as400, etc.) and its programs that are written in CL. How does one access the parameters within a given .PGM extension file (RPGLE) outside the application itself and without admin privs?
I have tried all available and related functions within the as400 app itself. None of them work. At least not with my user privs.
Overall, this mainframe app utilizes hundreds of libraries within libraries. Add thousands of files to that and you have a daunting task of accessing relevant data outside of the limited menu functions / programs(.PGM file extensions) for which you are granted access.
Frankly, the programs just suck. I really like using a SSH connection and qshell to execute SQL statements to bring up tables within the libraries. I can customize what tables are accessed and how the information is fetched.
This particular program acts as a basic query that accesses data from several tables, presents it in a new table that and allows for realtime filtering based on a limited amount of criteria.
I have tried the CAT command as below:
$ cat someprogam (and several variations with and without extensions, I forget)
But these just give me error codes
I have also tried executing SQL statements at the db2 prompt, but I get SQL STATE return codes (this isn't a .FILE extension so yes it makes sense that it wouldn't work the same way as selecting tables would).
I'd prefer accessing the data within qshell. Perhaps I am doing something wrong? Any help is greatly appreciated. Thank you
Qshell is an alternative operating environment to what I'll call native for lack of a better name. Qshell is similar to AIX but not 100% identical. Qshell does not map the output of native programs to stdout, so you can't access the output of a native program via pipes.
If a native program writes to a display, the only reasonable way to redirect that output is to screen scrape it, which can be done by writing your own tn5250 emulator, intercepting the screen output and putting it where you want it to go, including stdout. This is not particularly simple if you're not comfortable with the 5250 communications protocol.
You have two vaguely practical options. 1) Write many SQL stored procedures which, coupled together, replicate the business logic embedded within the programs you want to execute. 2) Ask the midrange developers to refactor out the business logic into a service program (they should understand this) and write a stored procedure for you to use. This way, both the midrange programs and external consumers (you) can reuse the same business logic. And when (not if) that logic changes, it'll change in one place, and all consumers will see the new rules.
Related
I am working using Tableau and have to write down multiple different SQL each time, while making new data sources.
I have to save all changes on SQL for every data source.
Currently I would paste the SQL on notepad and save them on separate folder in my computer, along with description of the changes.
Is there any better way to do this?
Assuming you have permission to create objects in the database, begin by creating database views, As #Nick.McDermaid commented.
Then, instead of using Custom SQL data source in Tableau, just connect to the View as if it were a table.
If you need to track the changes to these SQL views of your data, you will need to learn how to use source control for the .sql files that can be scripted from within SQL Server Management Studio:
Your company or school may have a preferred source control system already in use, in which case you should use that. If they don't, or if you are learning at home, then Git and Subversion are popular open source choices.
There are many courses available on learning platforms like Coursera that will teach you how to learn how to use those systems.
I had similar problem as you.
We ended up writing the queries in SQL Editor SQL Work bench (https://www.sql-workbench.eu/), then managed the code history and performed code peer-review (logic, error check, etc) in team shared space (like confluence).
The reasons we did that is
1) SQL queries are much easy to write on Work Bench
2) Code review is a must! You will find through implementing a review process more mistakes than you could ever think about
3) The shared space is just really convenient as it is accessible by everyone, and all errors are documented. After sometimes you get a lot of visible knowledge accumulated.
I also totally agree with Nick as this is one step to a reporting solution. But developing a whole reporting server is heavy, costly and takes time. Unless management are really convinced of the importance of developing a reporting solution, you may have to get a workaround with queries and Tableau (at least that was the case for us)
A little late to the party, but I would suggest you simply version the tableau workbook. The contents of the workbook are XML, so perfect for versioning using file based tools (Dropbox, One Drive, etc.) or source control (git, etc.). The workbooks themselves are usually quite small, so just make sure to keep the extract data separate if you use it.
Just a bit of background on where my question is coming from: my company has multiple databases across the globe that uses the same schema and once of my department's responsibility is to monitor and make sure all these DBs are in sync from a schema SQL change perspective.
Now, my question is if anyone knows of any Software/tool that has a a Frontend UI which is able to do the following (the lower number the more important to have):
Able to track what SQL code change was applied on which database and when. Basically, if we write a SQL query that changed the structure of a table and we need it applied to 80% or 100% percent of the DBs, either via manual input or some automatic check the tool will tell me that yes, this was indeed applied.
Code distribution tool: we give it the query or a file that contains the code and it's able to push to the Databases it needs to (and create the audit log for that)
Code/object repository: keeps track of what was custom developed and pushed to the databases
I know SSIS might be able to do some of these things, but we need a tool that also has a simple frontend interface that can be accessed by non-IT personnel. (*clarification: we are not planning on giving non-DBA people access to change things, just to the audit aspect of said tool)
I've tried searching the internet, but i have a feeling i'm not using the right vocabulary to get the results i'm looking for.
Hence i wanted to see if the community was aware of any such tool or something similar?
Try searching for one of these two types of systems:
Release/Build/Deployment Automation Complex programs like Serena that have modules for pushing, tracking, and auditing any kind of software, anywhere. These will include all the GUI bells and whistles. But you'll have to deal with extra databases, configuration, agents, workflows, consultants(?), etc. These programs are geared more towards developers.
Remote Execution/Configuration Management Simpler programs like Salt, Fabric, and Ansible that let you run operating system commands anywhere. They don't offer as many features, and you have to do more of the work yourself, but in some ways that's liberating. If you know exactly what commands you want to run you don't need some other program holding your hand. These programs are geared more towards administrators.
From a database administrator's point of view, the main problem with those types of programs is that none of them are relational. Yes they can connect to a database and run a script, but none of them really speak SQL. Their native languages are Java, XML, SSH, etc. There's nothing wrong with those technologies, but if you only care about databases you don't want to deal with all that complexity.
If you're not happy with either of those types of programs I recommend you look at my open source program Method5. It is a remote execution program built as an extension to Oracle SQL. It works entirely inside an Oracle database, so you can install it yourself and won't need any additional websites, agents, configuration files, GUIs, etc.
Based on your comment about getting bogged down by links, and my answer to your question about half a year ago, I think this is the kind of program you were gradually heading towards creating. It took my team a couple thousand hours of developing and testing to get it right so you were probably wise to give up on making your own.
To specifically answer your requirements:
Tracking Changes are stored in an audit trail. But more importantly it has the ability and a pre-built script to compare an unlimited number of schemas, all in one view. At the end of the day what you really want to know is "are my schemas the same", not necessarily "did the same thing get run everywhere?".
Code Distribution If you just have SQL or PL/SQL, deploying it through Method5 is as easy as it can possibly get. Just specify what you want to run, and where you want to run it, like this: select * from table(m5('create index ...', 'dev, qa, prodDB1, prodDB2')); The program does not (yet) run SQL*Plus scripts. But when you have the ability to run SQL and PL/SQL so easily there's little need for SQL*Plus.
Code Repository All executions are stored in a simple table, M5_AUDIT. It contains the code, who ran it, where they ran it, and how they ran it. It wasn't designed to be a repository like SVN but it's good enough for simple auditing and tracking code.
Method5 does not contain a GUI but in some ways I consider that to be a feature. Since everything is done relationally, everything is in a simple table. You can use any of your existing GUIs - Toad, PL/SQL Developer, Excel, Apex, etc. It's a robust back-end solution that will hopefully make a good foundation for easily building a simple front end.
I have a medium-sized app written in Ruby, which makes pretty heavy use of a RDBMS. As our code grows, I found the ugly SQL statements are spreading to all modules and methods in my app and embedded in many application logic. I am not sure if this is bad, however, my gut tells me this is quite ugly...
So generally in any languages, how do you manage your SQL statements? Or do you think it is harmful for maintainibility to let many SQL statements embedded in the application logic? Why or why not?
Thanks.
SQL is a language for accessing databases. Often, it gets confused as being the API into the data store for a larger application. In fact, you should design a real API between the data store and the app.
The means several things.
For accessing data stored in tables, you want to go through views in the database, rather than directly access the tables.
For data modification steps, you want to wrap insert/update/delete in stored procedures. This has secondary benefits, where you can handle constraints and triggers in the stored procedure and better log what is happening.
For security, you want to include database security as part of your security architecture. Giving all users full access may not be the best approach.
Unfortunately, it is easy to write a simple app that uses a database directly, whether in java or ruby or VBA or whatever. This grows into a bigger app, and then the maintenance problems arise.
I would suggest an incremental approach to fixing this. Go through the code and create views where you have nasty select statements. You'll probably find you need many fewer views than selects (the views can be re-used -- a good thing).
Find places where code is being modified, and change these to stored procedures. I always return status from the stored procedure for error checking and put log information into a table called someting like splog or _spcalls.
If you want to limit permissions for different users of your app, then you might be interested in this.
Leaving the raw SQL statements in the code is a problem. Just wait until you want to rename a column and you have to find all the places where this breaks the code.
Yes, this is not optimal - maintenance becomes a nightmare; it's hard to forecast and determine which code must change when underlying DB changes occur. This is why it is good practice to create a data access layer (DAL) to encapsulate CRUD operations from the application logic. There is often an business logic layer (BLL) between the application logic and DAL to enforce business rules/logic.
Google "data access layer" "business logic layer" and even "n-tier architecture" to learn more.
If you are concerned about the SQL statements littered around your application logic, maybe consider implementing them as Stored Procedures?
That way you will only be including the procedure name and any parameters that need to be passed to it in your code.
It has other benefits too, a common one being easier to re-use in multiple files.
There is much debate about speed and security of Stored Procedure and you will never get a definitive answer about that so I won't even open that can of worms.
Here is how you do this with Java: Create a class that encapsulates all access to the database. Add a method to the class for each query you need to run.
The answer for ruby will be similar to this.
It depends on the architecture of your application but a simple solution is to keep each sql in a file, qry.sql. For each Ruby module (or whatever is used in Ruby to aggregate related code) you can keep a folder SQL with these files. So, the collection of SQL folder/files form the data access layer of your application. The Ruby code provides the business layer. If your data model changes (field names, etc), you can do greps to identify the sql files that need changes. Anyway, definitely separate SQL from your logic code.
I had a (friendly but heated) argument with my lead developer the other day because our project has TSQL Scripts that I code directly into SQL files which I then run against the database. I find that when I do this, it's easy to work out the schema in advance without fiddly pointing and clicking and then there's no opportunity to forget to generate a script to put into source control as generating the script no longer becomes a chore you have to do after the fact, but is an implicit part of the process (and also leads to cleaner scripts without the extra crap that SQL Server Management Studio inserts into the scripts it generates).
My lead developer insists that having to manually script it out is a pain in the arse and that he absolutely refuses to write his scripts by hand when there are perfectly good tools to do it without coding. I've noticed that the copying of his changes into the actual scripts tends to get delayed a bit as a result though.
What are your thoughts on the pros and/or cons of doing it one way vs the other? Am I being too rigid/old-school in my sticking to hand coding schema scripts or is he being too reliant on third party tools and losing something in the process?
I always script stuff myself because the wizards sometimes don't script things in a way that I like it and will also give funky names to defaults
scripting things yourself is also good in case you get laid off and you have to go for an interview where they ask you to script DDL on the whiteboard
As I usually collaborate with a colleague during the schema design, I tend to design the schema using the GUI tools, as its easier to discuss it with a diagram of the tables in front of you. I then generate the scripts, being careful to select the exact options that I want to avoid having to make manual changes post-export.
I think a decision on the relative merits of the two approaches might take into account factors such as
the frequency of changes to the schema
the frequency with which changes need to be propagated to other schemas (test, user acceptance, production, clients * n, etc)
the degree to which the schema may vary across development branches
how well-known in advance your various changes can be scheduled
whether or not you can generate SQL "diff" scripts between schemas.
On balance, I tend to prefer to work with a script for each change (or "migration"). It lets me resequence change releases as priorities shift.
Just because you can create tables in a graphical tool doesn't necessarily mean you should.
I find its as quick to write a script as it is to use SQLMS. You still have to type names in SQLMS, and the time spent moving from keyboard and mouse could be used writing the proper script anyway.
The two of you are almost working with two sets of code. Consistency seems to be a key factor on these types of decisions. In your case, if you create a script, your boss uses the gui to add a field, how do you stay in sync? You can't use your script to rebuild the table without editing it (Chance for error.).
Maybe he should pull rank and force you to format your scripts the same way the GUI creates them - just kidding.
I think you should flip on it..........
We have a Delphi 2006 application that drives a MS SQL Server database.
We have found a vulnerability where it is possible to load the executable into a hex editor and modify the SQL.
Our long term plan is to move this SQL to CLR stored procedures but this is some way off since many of our clients still use SQL 2000.
We've thought about obfuscating the strings, does anyone have a recommendation for a tool for doing this?
Is there a better solution, maybe code signing?
Sorry for being blunt, but if you are thinking of applying "security" measures in your executable you are doomed. No scrambling schema will retain an average hacker.
You also haven't explained how is your app designed. Is the database hosted by you, or resides in your client's premises? If the latter, then just forget about security and start hiring a lawyer to get a good confidentiality contract so your clients behave. If the former, then using stored procedures is the easiest way.
If embedded SQL is being hacked, then it implies that your database is quite open and anyone with MSQRY32.EXE (that is, MS Office) can get your data.
If you are a vendor, then you can't rely on CLR being enabled at your clients. So, why not use non-CLR stored procedures and correct permissioning in the database that is version independent?
This is not a vulnerability. If your machines are vulnerable to having people locally modify EXEs, that is your vulnerability.
All EXEs can be hacked, if someone has local admin account access, your game is over long before they get near your resource strings.
It will never be possible to protect completely, but you can make "casual attack" harder. The simple system that I use is a "ROT47" type system which is like ROT13 but wider ranging. The code then gets to look like the following:
frmLogin.Caption := xIniFile.ReadString(Rot47('$JDE6>' {CODEME'System'}),
The key here is that I have a comment which includes the string so both I can see it, but more importantly so can the utility that I run in my FinalBuilder build script. This allows me to ensure that strings are up-to-date at all times in release code. The utility looks for {CODEME in the lines, and if found knows the format of the data to output appropriately.
A solution that would require a deep restructuring of the application would be to use a multi-tier approach - most the of the SQL code would be in the application server module, that being on a server should be more protected than a client side exe.
Can't you encrypt all your queries and put them to the resource file?
During runtime, firstly you would have to:
Load your query string from resource.
Decrypt it.
Then you just run your query as before.
That should not be a big problem. Of course if you are not storing your queries in some resource / folder than you need to refactor your application a bit. But you should store them anyway in some organized manner. So you will be hitting a two birds with one stone here ;-)
For encryption of the strings you could use a free library called DCPCrypt.
I think you should use a exe packer which makes it hard for anyone to modify the stuff using hex editor.
First - do an analysis of your threat. Who is using your vulnerability, why is this a problem. Then act accordingly.
If your application is win32 and your threat are some kids witch are just having fun, a free exe packer (e.g. upx) might be the solution. On .NET applications signing might be what you want.
If you need more than that, it's going to be expensive and it's going to be more difficult to develop your application. Perhaps you even need to restructure it. Commercial protection schemes are available (perhaps with dongle?) - even protection schemes where you store your strings on some external hardware. If the hardware is not present, no SQL-Strings. But, as I said, that's more expensive.
Move DB interface to stored procedures. Normal regular stored procedures without any CLR. It's not a big deal if you already have queries to put inside.
If you don't want to learn T-SQL for some reasons, simple move all you query string to database and store in application single query, which purpose is reading SQL code with given query ID from database only.
All tricks with encoding produces a lot of troubles, but don't give any real security because must use reversable encrypting (dictated by the nature of the problem) and all keys for decoding placed in application executable too.
There are "protection" suites that encrypt and/or validate your exe before running. searching for "encrypt exe" or "validate exe" or so will probably help. Usually they are payware, but sub $100.
The principle is the same as an exe packer (and has some of its downsides, like cheaper antivirus heuristics sometimes reacting on them, a slightly elevated memory load), just more focussed on security. A problem is also that for most exe packers, depackers exist.
I use dinkeydongle's wares, but that is a kind that also ties into an hardware dongle, so that might be a bridge to far for you.