Today I am being asked to format long SAS script with mainly Proc SQL which are not readable (do not respect simple SQL rules of readability):
imbricated SQL queries with no indentation
case is not respected
etc...
I tried automatic SaS formatter but it do not format Proc SQL. Do you have any ideas ? We have many scripts and the Team is ready to do that manually, it seems prone to error and I am not sure we'll have the same syntax at the end.
Any tips would be welcome!
I can add code snippets if needed but I think that the problem is clear and I am not the first to encounter it.
I would suggest ignoring the fact that you're in SAS for the moment, and instead focus on the SQL itself. Find a language you're comfortable with that has libraries that format code in other languages - Python for example can do this - and then:
Open the .sas file as a text file
Find "PROC SQL" text and grab from there to the "QUIT" (case insensitive)
Pass that inner text to the SQL code formatter
Grab the result and insert it back into the text file
Something along those lines is your best bet. SAS doesn't have anything built-in for this, so you're going to have to go outside here.
Can anybody tell me what does the error mean? Whenever I open the query builder it will prompt with an error indicating that SQL syntax errors were encountered.
https://msdn.microsoft.com/en-us/library/ms189012.aspx
I looked at the following page in MSDN but I don't understand what it means...
For instance, what do these bullet points from the MSDN article mean?
The SQL statement is incomplete or contains one or more syntax errors.
The SQL statement is valid but is not supported in the graphical panes (for example, a Union query).
The SQL statement is valid but contains syntax specific to the data connection you are using.
USER (which you've apparently decided is an appropriate table name) is a SQL Server reserved word.
The best solution is to rename your table, so you don't have to escape the table name every time you want to query it and to make it clear it's your user data (hey, there's a table name suggestion - userdata).
The other option is to escape the name by surrounding it with square brackets:
SELECT * FROM [users]
Note that it will get old fast having to do this with every query. Again, the best solution would be to rename the table to something that isn't a reserved word.
I have the following problem, i have a SQL file to execute with DBI CPAN module Perl
I saw two solution on this website to solve my problem.
Read SQL file line by line
Read SQL file in one instruction
So, which one is better, and what the real difference between each solution ?
EDIT
It's for a library. I need to retrieve output and the return code.
Kind of files passed might be as following:
set serveroutput on;
set pagesize 20000;
spool "&1."
DECLARE
-- Récupération des arguments
-- &2: FLX_REF, &3: SVR_ID, &4: ACQ_STT, &5: ACQ_LOG, &6: FLX_COD_DOC, &7: ACQ_NEL, &8: ACQ_TYP
VAR_FLX_REF VARCHAR2(100):=&2;
VAR_SVR_ID NUMBER(10):=&3;
VAR_ACQ_STT NUMBER(4):=&4;
VAR_ACQ_LOG VARCHAR2(255):=&5;
VAR_FLX_COD_DOC VARCHAR2(30):=&6;
VAR_ACQ_NEL NUMBER(10):=&7;
VAR_ACQ_TYP NUMBER:=&8;
BEGIN
INSERT INTO ACQUISITION_CFT
(ACQ_ID, FLX_REF, SVR_ID, ACQ_DATE, ACQ_STT, ACQ_LOG, FLX_COD_DOC, ACQ_NEL, ACQ_TYP)
VALUES
(TRACKING.SEQ_ACQUISITION_CFT.NEXTVAL, ''VAR_FLX_REF'',
''VAR_SVR_ID'', sysdate, VAR_ACQ_STT, ''VAR_ACQ_LOG'',
''VAR_FLX_COD_DOC'', VAR_ACQ_NEL, VAR_ACQ_TYP);
END;
/
exit;
I have another question to ask, again with DBI Oracle module.
May i use the same code for SQL file and for Control file ?
(Example of SQL Control file)
LOAD DATA
APPEND INTO TABLE DOSSIER
FIELDS TERMINATED BY ';'
(
DSR_IDT,
DSR_CNL,
DSR_PRQ,
DSR_CEN,
DSR_FEN,
DSR_AN1,
DSR_AN2,
DSR_AN3,
DSR_AN4,
DSR_AN5,
DSR_AN6,
DSR_PI1,
DSR_PI2,
DSR_PI3,
DSR_PI4,
DSR_NP1,
DSR_NP2,
DSR_NP3,
DSR_NP4,
DSR_NFL,
DSR_NPG,
DSR_LTP,
DSR_FLF,
DSR_CLR,
DSR_MIM,
DSR_TIM,
DSR_NDC,
DSR_EMS NULLIF DSR_EMS=BLANKS "sysdate",
JOB_IDT,
DSR_STT,
DSR_DAQ "CASE WHEN :DSR_DAQ IS NOT NULL THEN SYSDATE ELSE NULL END"
)
Reading a table one row at a time is more complex, but it can use less memory - provided you structure your code to make use of the data per item and not need it all later.
Often you want to process each item separately (e.g. to do work on the data), in which case you might as well use the read line-by-line approach to define your loop.
I tend to use single-instruction approach by default, but as soon as I am concerned about number of records (especially in long-running batch processes), or need to loop through the data as the first task, then I read records one-by-one.
In fact, the two answers you reference propose the same solution, to read and execute line-by-line (but the first is clearer on the point). The second question has an optional answer, where the file contains a single statement.
If you don't execute the SQL line-by-line, it's very difficult to trap any errors.
"Line by line" only makes sense if each SQL statement is on a single line. You probably mean statement by statement.
Beyond that, it depends on what your SQL file looks like and what you want to do.
How complex is your SQL file? Could it contain things like this?
select foo from table where column1 = 'bar;'; --Get foo; it will be used later.
The simple way to read an SQL file statement by statement is to split by semicolons (or whatever the statement delimiter is). But this method will fail if you might have semicolons in other places, like comments or strings. If you split this statement by semicolons, you would try to execute the following four "commands":
select foo from table where column1 = 'bar;
';
--Get foo;
it will be used later.
Obviously, none of these are valid. Handling statements like this correctly is no simple matter. You have to completely parse SQL to figure out what the statements are. Unfortunately, there is no ready-made module that can do this for you (SQL::Script is a good start on an SQL file processing module, but according to the documentation it just splits on semicolons at this point).
If your SQL file is simple, not containing any statement delimiters within statements or comments; or if it is predictable in some other way (such as having one statement per line), then it is easy to split the file into statements and execute them one by one. But if you have to handle arbitrary SQL syntax, including cases such as above, this will be a complex task.
What kind of task?
Do you need to retrieve the output?
Is it important to detect errors in any individual statement, or is it just a batch job that you can run and not worry about it?
If this is something that you can just run and forget about, you could just have Perl execute a system command, telling Oracle to process the file. This will be simpler than handling all of the statements yourself. But if you need to process the results or handle errors within Perl, doing it yourself statement by statement will be a necessity.
Update: based on your response, you want to write a library that can handle arbitrary SQL statements. In that case, you definitely need to parse the SQL and execute the statements one at a time. This is do-able, but not simple. The possibility of BEGIN...END blocks means that you have to be able to correctly handle semicolons within a statement.
The SQL::Statement class of modules may be helpful.
Context
We're changing our install scripts to use ant's "sql" task and jdbc rather than proprietary sql clients sqlplus (oracle) and osql (msft).
Updated: added more context. Our "base data" (seed data) consists of a collection of .sql files containing "vendor-neutral"(i.e. works both in oracle and mssql) sql statements.
The Problem
The scripts run fine, with one exception:
This sql fails in Oracle. Specifically, something (ant or jdbc driver) treats the dashes/hyphens as "beginning of a comment"--even though they are embedded in a string. Note that the same sql works fine with ant/sql and microsoft's jdbc driver.
INSERT INTO email_client (email_client_id,generated_reply_text) VALUES(100002,'----- Original Message -----');
Related Bug
This ant bug appears to identify the problem. As it's still open (after 8 years), I'm not hoping for a fix soon. However, because the problem appears only in oracle, it may lie with the driver.
The oracle driver: jdbc thin driver, version 10.2.0.1.0
The Question
Does anyone have a workaround which works in both mssql and oracle? (e.g. changing the offending lines to define an escape character? I don't see an 'escape' on the 'insert' sql92 syntax)
thanks
After viewing the 'SQLExec' source and turning on verbose logging, I found a workaround:
Workaround
if the sql statement includes a string containing '--', place the delimiter (semi-colon) on the next line.
This Fails
INSERT INTO email_client (email_client_id,generated_reply_text) VALUES(100002,'----- Original Message -----');
This Succeeds
Note that semi-colon is on a separate line
INSERT INTO email_client (email_client_id,generated_reply_text) VALUES(100002,'----- Original Message -----')
;
Details
Turning on verbose logging, I saw that when Ant came across the offending sql statement, it actually passed three sql statements in at once to the jdbc driver. The offending statement, the next statement (which also included an embedded '--'), and the subsequent statement (which did not include an embedded '--').
I gave the Ant code a quick glance and didn't see any obvious errors. Since I wasn't planning to patch Ant, I looked for a workaround.
Tweaking with it I found that if I simply moved the delimiter (semicolon) to the next line for the statements with embedded '--', the scripts executed successfully.
thanks everyone for weighing in
You could try this:
INSERT INTO email_client (email_client_id,generated_reply_text)
VALUES(100002,LPAD('-',5,'-') || ' Original Message ' || LPAD('-',5,'-'));
I have been researching a way to get the SQL statements that are built by a generated Migration file. These extend Doctrine_Migration_Base. Essentially I would like to save the SQL as change scripts.
The execution path leads me to Doctrine_Export which has methods that build the SQL statement and executes them. I have found no way of asking for just them. The export methods found in Doctrine_Export only operate on Doctrine_Record models and not Migration scripts.
From the command line './doctrine migrate version#' the path goes:
Doctrine_Cli::run(cmd)
Doctrine_Task_Migrate::setArguments(args)
Doctrine_Task_Migrate::execute()
Doctrine_Migration::migrate(to)
Doctrine_Migration_Process::Doctrine_Export::various
create, drop, alter methods with sql
equivalents.
Has anyone tackled this before? I really would not like to change Doctrine base files. Any help is greatly appreciated.
Could you make a dev server, and do the migration on that, storing a SQL Trace as you go?you don't have to keep the results, but you would get a list of every command.
Taking into account Rob Farley's suggestion, I modified:
Doctrine_Core::migrate
Doctrine_Task_Migrate::execute
When the execute method is called the optional argument 'dryRun' is checked. If true
then a 'Doctrine_Connection_Profiler' instance is created. The 'dryRun' value is then passed onto
the 'Doctrine_Core::migrate' method. The 'dryRun' value of true allows the changes to rollback when done executing the SQL statements. When the method returns, the profiler is parsed and non-empty SQL statements
not containing 'migration_version' are saved and displayed to the terminal.