perl execute sql file (DBI oracle) - sql

I have the following problem, i have a SQL file to execute with DBI CPAN module Perl
I saw two solution on this website to solve my problem.
Read SQL file line by line
Read SQL file in one instruction
So, which one is better, and what the real difference between each solution ?
EDIT
It's for a library. I need to retrieve output and the return code.
Kind of files passed might be as following:
set serveroutput on;
set pagesize 20000;
spool "&1."
DECLARE
-- Récupération des arguments
-- &2: FLX_REF, &3: SVR_ID, &4: ACQ_STT, &5: ACQ_LOG, &6: FLX_COD_DOC, &7: ACQ_NEL, &8: ACQ_TYP
VAR_FLX_REF VARCHAR2(100):=&2;
VAR_SVR_ID NUMBER(10):=&3;
VAR_ACQ_STT NUMBER(4):=&4;
VAR_ACQ_LOG VARCHAR2(255):=&5;
VAR_FLX_COD_DOC VARCHAR2(30):=&6;
VAR_ACQ_NEL NUMBER(10):=&7;
VAR_ACQ_TYP NUMBER:=&8;
BEGIN
INSERT INTO ACQUISITION_CFT
(ACQ_ID, FLX_REF, SVR_ID, ACQ_DATE, ACQ_STT, ACQ_LOG, FLX_COD_DOC, ACQ_NEL, ACQ_TYP)
VALUES
(TRACKING.SEQ_ACQUISITION_CFT.NEXTVAL, ''VAR_FLX_REF'',
''VAR_SVR_ID'', sysdate, VAR_ACQ_STT, ''VAR_ACQ_LOG'',
''VAR_FLX_COD_DOC'', VAR_ACQ_NEL, VAR_ACQ_TYP);
END;
/
exit;
I have another question to ask, again with DBI Oracle module.
May i use the same code for SQL file and for Control file ?
(Example of SQL Control file)
LOAD DATA
APPEND INTO TABLE DOSSIER
FIELDS TERMINATED BY ';'
(
DSR_IDT,
DSR_CNL,
DSR_PRQ,
DSR_CEN,
DSR_FEN,
DSR_AN1,
DSR_AN2,
DSR_AN3,
DSR_AN4,
DSR_AN5,
DSR_AN6,
DSR_PI1,
DSR_PI2,
DSR_PI3,
DSR_PI4,
DSR_NP1,
DSR_NP2,
DSR_NP3,
DSR_NP4,
DSR_NFL,
DSR_NPG,
DSR_LTP,
DSR_FLF,
DSR_CLR,
DSR_MIM,
DSR_TIM,
DSR_NDC,
DSR_EMS NULLIF DSR_EMS=BLANKS "sysdate",
JOB_IDT,
DSR_STT,
DSR_DAQ "CASE WHEN :DSR_DAQ IS NOT NULL THEN SYSDATE ELSE NULL END"
)

Reading a table one row at a time is more complex, but it can use less memory - provided you structure your code to make use of the data per item and not need it all later.
Often you want to process each item separately (e.g. to do work on the data), in which case you might as well use the read line-by-line approach to define your loop.
I tend to use single-instruction approach by default, but as soon as I am concerned about number of records (especially in long-running batch processes), or need to loop through the data as the first task, then I read records one-by-one.

In fact, the two answers you reference propose the same solution, to read and execute line-by-line (but the first is clearer on the point). The second question has an optional answer, where the file contains a single statement.
If you don't execute the SQL line-by-line, it's very difficult to trap any errors.

"Line by line" only makes sense if each SQL statement is on a single line. You probably mean statement by statement.
Beyond that, it depends on what your SQL file looks like and what you want to do.
How complex is your SQL file? Could it contain things like this?
select foo from table where column1 = 'bar;'; --Get foo; it will be used later.
The simple way to read an SQL file statement by statement is to split by semicolons (or whatever the statement delimiter is). But this method will fail if you might have semicolons in other places, like comments or strings. If you split this statement by semicolons, you would try to execute the following four "commands":
select foo from table where column1 = 'bar;
';
--Get foo;
it will be used later.
Obviously, none of these are valid. Handling statements like this correctly is no simple matter. You have to completely parse SQL to figure out what the statements are. Unfortunately, there is no ready-made module that can do this for you (SQL::Script is a good start on an SQL file processing module, but according to the documentation it just splits on semicolons at this point).
If your SQL file is simple, not containing any statement delimiters within statements or comments; or if it is predictable in some other way (such as having one statement per line), then it is easy to split the file into statements and execute them one by one. But if you have to handle arbitrary SQL syntax, including cases such as above, this will be a complex task.
What kind of task?
Do you need to retrieve the output?
Is it important to detect errors in any individual statement, or is it just a batch job that you can run and not worry about it?
If this is something that you can just run and forget about, you could just have Perl execute a system command, telling Oracle to process the file. This will be simpler than handling all of the statements yourself. But if you need to process the results or handle errors within Perl, doing it yourself statement by statement will be a necessity.
Update: based on your response, you want to write a library that can handle arbitrary SQL statements. In that case, you definitely need to parse the SQL and execute the statements one at a time. This is do-able, but not simple. The possibility of BEGIN...END blocks means that you have to be able to correctly handle semicolons within a statement.
The SQL::Statement class of modules may be helpful.

Related

How to format SAS script with complex proc SQL

Today I am being asked to format long SAS script with mainly Proc SQL which are not readable (do not respect simple SQL rules of readability):
imbricated SQL queries with no indentation
case is not respected
etc...
I tried automatic SaS formatter but it do not format Proc SQL. Do you have any ideas ? We have many scripts and the Team is ready to do that manually, it seems prone to error and I am not sure we'll have the same syntax at the end.
Any tips would be welcome!
I can add code snippets if needed but I think that the problem is clear and I am not the first to encounter it.
I would suggest ignoring the fact that you're in SAS for the moment, and instead focus on the SQL itself. Find a language you're comfortable with that has libraries that format code in other languages - Python for example can do this - and then:
Open the .sas file as a text file
Find "PROC SQL" text and grab from there to the "QUIT" (case insensitive)
Pass that inner text to the SQL code formatter
Grab the result and insert it back into the text file
Something along those lines is your best bet. SAS doesn't have anything built-in for this, so you're going to have to go outside here.

How to run thousands of queries in Ms Access?

I've discovered the SQL VIEW in Ms Access to execute some queries, but I need to execute about 20.000 UPDATE queries I have in a .sql file.
When I paste in the SQL VIEW it says the "Text is too long to modify".
How can I run those UPDATE's ?
The limit to the number of characters in an Access SQL query is "about 64000" - see here https://support.office.com/en-us/article/Access-2010-specifications-HA010341462.aspx. And unfortunately you cannot execute multiple statements in a query. I think this will mean quite a bit of work for you in VBA. Here is an example approach (pseudocode):-
open file
read line into variable
while not EOF
currentdb.execute variable, dbfailonerror
read next line
wend
close file
Probably a nasty surprise for you if you are used to executing huge batches of statements using other RDBMS!
An alternative suggestion: we don't know exactly what your file looks like or where it comes from, but if it is generated from another RDBMS which you have access to, then I would very strongly recommend that you set up an ODBC connection to it, and query out the data you need (either by linking the tables or writing a pass through query), then inserting into your local Access tables. This will be many orders of magnitude faster than executing thousands of individual statements.
If your only source of the data is the SQL statements then you may still be better of if you can parse the SQL text into relevant columns (for example PK, and value to be updated, or if inserting, then all column values), then save as a csv file, import into Access, add keys as necessary, and then run a single update statement as an updateable query against the imported data and the existing tables. Dumping the file into Excel and using the various string functions may enable you to parse the data quite quickly.
There may be an easier way but you could write VBA code that reads the text file line by line and then uses DoCmd.RunSQL to run each query.

Is the semicolon necessary in SQL?

Sometimes it works anyway if I forget the ;. But sometimes it doesn't.
And in JDBC and Android SQLite, it seems that I don't need ; at all. I am confused.
When should I use a semicolon?
semicolon indicates end of a statement, so if there are multiple statements then you should use semicolon else it will work fine.
I generally use semicolon as a practice, it can be useful even when you are running queries on sql client e.g. in Sql Developer using semicolon is very helpful if you have multiple statements on worksheet, as you can simply go to that particular statement and use F9 to execute that, without semicolon this is not possible.
It is not mandatory if you run a single query at time, it comes necessary instead if you want to run multiple query with a single command.
However in most of JDBC drivers out there it is not possible to add multiple query separated with semicolon in a single JDBC Command, it exist however the addBatch method that allow you to add multiple statements :
java.sql.Statement stmt=con.createStatement();
stmt.addBatch(insert_query1); //insert_query1
stmt.addBatch(insert_query2); //insert_query2
As a rule of thumb, in JDBC semicolon is not necessary at all, if you need multiple statement use addBatch.
Usually the semicolon is not part of the actual syntax of a statement (as most database internal APIs execute a single statement at a time). Instead the semicolon is an 'end-of-statement' marker or statement separator that is - usually - defined in CLI or scripting tools for the database. This allows that tool to know when a statement ends, so it can send that single statement to the database for execution.
On the other hand, the JDBC API is intended to execute a single(!) statement at a time, therefore you don't need such a separator (the statement is the whole string). This means that a semicolon is not needed, and as it is not part of the actual statement syntax for a lot of database it is also a syntax error to include it. Some JDBC drivers will strip the last ; from a statement to 'fix' that, some drivers don't.
Some drivers allow - contrary to the JDBC specification - multiple statements to be executed as a single string, this usually has to be enabled with a connection property, for example for MySQL it is the option allowMultiQueries (see the MySQL properties for details).
Depends on the DBMS and version number. Semicolons are often optional at the end of a single statement. But if you are going to execute a script with more than one statement, they need to be terminated by a semicolon.
Except maybe the last one. But it seems bad form to be inconsistent.

How do I programmatically run a complex query on an as400?

I'm new at working on an as400 and I have a query the joins across 4 tables. The query itself is fine, it runs in STRSQL and displays the results.
What I am in struggling with is getting the query to be able to run programmatically (it will eventually be run from a scheduled CL script).
I tried have creating a physical file that contains the query running it with RUNQRY, but it simply displays the query itself, not the actual result set.
Does anyone know what I am doing wrong?
UPDATE
Thanks everyone for the direction and the resources, with them I was able to reach my goal. In case it helps anyone, this is what I ended up doing (all of this was done in it's own library, ALLOCATE):
Created a source physical file (using CRTSRCPF): QSQLSRC, and created a member named SQLLEAGSEA, with the type of TXT, that contains the SQL statement.
Created another source physical file: QCLSRC, and created a member named POPLEAGSEA, with the type of CLP, that changes the current library to ALLOCATE then runs the query using RUNSQLSTM (more detail on this below). Here is the actual command:
RUNSQLSTM SRCFILE(QSQLSRC) SRCMBR(SQLLEAGSEA) COMMIT(*NONE) NAMING(*SYS)
Added the CLP to the scheduled jobs (using ADDJOBSCDE), running the following command:
CALL PGM(ALLOCATE/POPLEAGSEA)
With regard to RUNSQLSTM, my research indicated that I wasn't going to be able to use this function, because it didn't support SELECT statements. What I didn't indicate in my question was what I needed to do with the the result - I was going to be inserting the resultant data into another table (had I done that I'm sure the help could have figured that out a lot quicker). So effectively, I wasn't going to be doing an SELECT, my end result is actually an INSERT. So my SQL statement (in SQLLEAGSEA) begins with:
INSERT INTO
ALLOCATE/LEAGSEAS
SELECT
...
BLAH BLAH BLAH
...
From my research, I gather that RUNSQLSTM doesn't support SELECT because it doesn't have a mechanism to do anything with the results. Once I stopped taking baby steps and realized I needed to SELECT AND INSERT in the same statement, it solved my main problem.
Thanks again everyone!
The command is RUNSQLSTM to run a static SQL statement in a physical file member or stream file.
It is a non-interactive command so it will not execute sql statements that attempt to return a result set.
If you want more control, including the ability to run interactive statements, see the Qshell db2 utility.
For example:
QSH CMD('db2 -f /QSYS.LIB/MYLIB.LIB/MYSRCFILE.FILE/MYSQL.MBR')
Note that the db2 utility only accepts the *SQL naming convention.
QM Query
If all the SQL you need is the single complex SQL statement, and this is what it sounds like, then your best bet is to use Query Management Query (see QM Query manual here).
The results can be directed to a display, a spool file, or a physical file (ie a DB2 table). The default output when run interactively is to the screen, but when run in a (scheduled) batch job it will default to a spool file report.
You can create the QM Query interactively via WRKQMQRY, in prompted mode (much like Query/400) or in SQL mode. Or you can compile the QM Query from source, with the CRTQMQRY command.
To run your QM Query, STRQMQRY command.
RUNSQL cmd
If you are using a system that has IBM i 7.1 fully up-to-date, and has Technology Refresh 4 (TR4) installed, then you could also use the new RUNSQL command to execute a single statement. (see discussion in developerWorks)
SQL Scripting w/ RUNSQLSTM cmd
From CL you can run SQL scripts of multiple SQL statements from a source file member. There is no standard default source file name for this, but QSQLSRC is commonly used. The source member can contain multiple non-interactive SQL statements. This means you cannot use a SELECT statement (directly) since theoretically it will not know where to send the results. CL commands are even allowed if given a CL: prefix. Both SQL and CL statements should be terminated with a semicolon ;. While the SQL statements cannot display data directly to the screen, the same restriction does not apply to the scripted CL commands.
The STRQMQRY command can be embedded in the RUNSQLSTM script, by placing the prefix "CL: " in front of the command. Since STRQMQRY can direct output to the screen, a report, or an output table, this can come in very useful.
Remember that to direct your output from a SELECT query to a file you can use either the INSERT or CREATE TABLE statements.
CREATE TABLE newtbl AS
( full-select )
WITH DATA;
Or, to put the results into a table you create in your job's QTEMP library:
DECLARE GLOBAL TEMPORARY TABLE AS
( full-select )
WITH DATA;
[Note: If you create the source to be used by CRTQMQRY, you are advised to create it as CRTSRCPF yourlib/QQMQRYSRC RCDLEN(91), since the compiler will only use 79 columns of your source data (adding 12 for sequence and change date =91). However for QM Forms, which can be used to provide additional formatting, the CRTQMFORM compiler will use 81 columns so RCDLEN(93) is advised for QQMFORMSRC.]
RUNQRY is a utility that lets you execute a query that was created by another utility named WRKQRY. If you really want to process SQL statements held in a file try RUNSQLSTM. It uses a source physical file to store the statements, not a database file. The standard name for that source physical file is QQMQRYSRC. To create that file, CRTSRCPF yourlib/QQMQRYSRC. Then you can use PDM to work with that source PF. WRKMBRPDM yourlib/QQMQRYSRC. Use F6 to create a new source member. Make it source type TXT. Then use option 2 to will start an editor called SEU. Copy/paste your SQL statements into this editor. F3 to save the source. Once the source is saved, use RUNSQLSTM to execute it.
It is (now) possible to run SQL directly in a CL program without using QM Query, RUNSQLSTM or QShell.
Here is an article that discusses the RUNSQL statement in CL programs...
http://www.mcpressonline.com/cl/the-cl-corner-introducing-the-new-run-sql-command.html
The article contains information on what OS levels are supported as well as clear examples of several ways to use the RUNSQL statement.
This will work in two steps:
RUNSQL SQL('CREATE TABLE QTEMP/REPORT AS (SELECT +
EXTRACT_DATE , SYSTEM, ODLBNM, SUM( +
OBJSIZE_MB ) AS LIB_SIZE FROM +
ZSYSCOM/DISKRPTHST WHERE ODLBNM LIKE +
''SIS%'' GROUP BY EXTRACT_DATE, SYSTEM, +
ODLBNM ORDER BY LIB_SIZE DESC) WITH +
DATA') COMMIT(*NONE) DATFMT(*USA) DATSEP(/)
RUNQRY QRYFILE((QTEMP/REPORT)) OUTTYPE(*PRINTER) +
OUTFORM(*DETAIL) PRTDFN(*NO) PRTDEV(*PRINT)
The first step creates a temporary table result in qtemp and the second step/line runs an adhoc query over just the temporary table to a spool file.
Thanks,
Michael Frilot
There is of course a totally different solution: You could write and compile a program containing the statement. It requires some longer reading into, especially if you are new to the platform, but it should give you most flexibility over what you do with results. You can use SQL in C, C++, RPG, RPG/LE, REXX, PL (of which I don't know, what it is) and COBOL. Doing that, you can react in any processable way on results from one query and start/create other queries based on what you get.
Although some oldfashioned RPG-programmers try everything to deny SQL in RPG exists, it is possible today for many cases, to write RPG-programs with SQL only and no direct file access (without F-Specs, for those who know RPG).
If your solution works for you, perfect. If you need to do something else, try a look into this pdf: http://publib.boulder.ibm.com/infocenter/iseries/v5r3/topic/rzajp/rzajp.pdf
The integration into RPG is not too bad. It works with the normal program flow. Would look something like this (in free form):
/free
// init search values:
searchval = 'Someguy';
// so the sql query:
exec sql
SELECT colum1, colum2
INTO :var1, :var2
FROM somelib/somefile
WHERE keycol=:searchval;
// now do something with the values:
some_proc(var1);
/end-free
In this, var1, var2, and searchval are ordinary RPG-variables. No quoting needed. Works also with datastructures (externally defined e.g., the record format of the file itself fits well). You can work with cursors and loops, too, of course. I feel that RPG-programs tend to be easier to read with this.

Removing unwanted SQL queries based on a condition

I have not had experience in SQL queries or SQL database , so please excuse me if my terminology is wrong.
So, I have a file containing around 17,000 SQL insert statements where I enter data for 5 columns/attributes in a database. In those 17,000 statements there are only around 1200 statements which have data for all of the 5 columns/attributes while the rest have data only for 4 columns. I need to delete all those unwanted statements( which dont have data for all 5 columns).
Is there a simple way/process to do it other than going one by one and deleting? If so, it would be great if someone could help me out with it.
A different approach from my fine colleagues here would be to run the file into a staging/disposable database. Use the delete that #Rob called out in his response to pare the table down to the desired dataset. Then use an excellent, free tool like SSMS Tools Pack to reverse engineer those insert statements.
I can think of two approaches:
1: Using SQL: insert all the data and then run a query that removes any records where it does not have all of the necessary data. If the table is not currently empty, keep track of the ID where your current data "ends" so that your query can use that as a WHERE statement.
DELETE FROM myTable WHERE a IS NULL OR b IS NULL /* etc. */
2: Process the SQL file with a regular expression: Use a text editor or command line to match either "bad" records or "good" records. Most text editors have a find and replace that allows you to use regular expressions. And command line you can use grep or other tools to process. Or even a script that parses in your language of choice, for that matter.
Open file in notepad++, replace all "bad" lines using regular expressions.