I have the following problem:
I have a table Source_Data(actually, quite many different source tables), and I need to export some data from them into Result_Table. For each source table I have some sql that returns the data, and it needs to be transformed into csv-like string.
For example, Source_Data1 table has PK - ID, some foreign keys - A_ID, B_ID, CREATION_DATE date, and some other columns.
I need to export the result of SQL query that selects data from Source_Data1 table, it usually returns the primary key, some foreign keys which are not null and a date which is also not null, and transform it into cvs-like string, which should be saved later in some other table as a clob.
CREATE TABLE Source_Data1 (
ID NUMBER(3) NOT NULL,
A_ID NUMBER(10) NOT NULL,
B_ID NUMBER(10) NOT NULL,
CREATION_DATE DATE NOT NULL,
some other columns );
and I have SQL query:
select ID, A_ID, B_ID, CREATION_DATE
from Source_Data1
where <expression>
Based on this query, I need to generate string like this:
'ID,A_ID,B_ID,CREATION_DATE
1,200,10,2018-03-01 00:00:00.0
7,202,11,2018-03-02 00:00:00.0
8,205,11,2018-03-02 00:00:00.0'
and I need to save it into the Result table as a clob:
Insert into Result_Table (ID, SOURCE_NAME, DATA) values
(result_seq.nextval, 'Source_Data1', result of sqlquery 1);
And this needs to be done for many source tables.
I was wondering if there is some general way to do it? Because for each source table I have its own sql query, different set of selected columns, and generating this csv string manually looks not very nice.
Thank you for your advice.
Step 1 should be to download SQLcl, you can easily save the output of SQL select statements in CSV format via "SET SQLFORMAT CSV" and SPOOL, it does the work for you. So you can use that to loop through a bunch of table names / select statements, spool and save the output of each to a directory via a shell script.
Next, here's a package that has a bunch of file utilities that all work with oracle DIRECTORY objects and the files within them. With this and some PL/SQL, you could easily pull those files that you've saved off into a table. There may be easier ways to do what you're trying to do, and if there is, I look forward to hearing about them for others. It's kind of a big problem to solve.
1) file_into_blob - To get the file into the database
2) convert_blob_to_clob - To convert to clob
3) You can then insert that into your table.
Package spec
Create or replace package fileutils as
--
-- This procedure deletes a file, and depends on an Oracle DIRECTORY object being passed
--
Procedure delete_os_file (i_directory varchar2, i_filename varchar2);
--
-- This procedure moves and optionally renames a file,
-- and depends on an Oracle DIRECTORY object being passed
--
Procedure move_os_file ( i_source_directory in varchar2, i_source_file in varchar2, i_target_directory in varchar2, i_target_file in varchar2);
--
-- This procedure takes a blob variable and writes it to a file,
-- and depends on an Oracle DIRECTORY object being passed
--
Procedure blob_into_file (i_directory in varchar2, i_file_name in varchar2, i_blob in blob);
--
-- This procedure takes a file and uploads it into a blob variable
-- and depends on an Oracle DIRECTORY object being passed
--
Procedure file_into_blob(i_directory in varchar2, i_file_name in varchar2, o_blob out blob);
--
-- This procedure converts a clob to a blob
--
Procedure convert_clob_to_blob (i_clob in clob, o_blob out blob);
--
-- This procedure converts a blob to a clob
--
Procedure convert_blob_to_clob (i_blob in blob, o_clob out clob);
--
-- This one checks for file existence without Java
--
Function file_exists (i_directory in varchar2, i_filename in varchar2) return boolean;
--
-- Returns the basename of a filename
-- Works with Windows and UNIX pathnames
--
Function basename (i_filename in varchar2) return varchar2;
--
-- This takes a Base64 string and converts it to a binary BLOB
--
Procedure base64_string_to_blob (i_clob in clob, o_blob out blob);
Function base64_string_to_blob (i_clob in clob) return blob;
--
-- This takes a binary BLOB and converts it to a Base64 string
--
Procedure blob_to_base64_string (i_blob in blob, o_clob out clob);
Function blob_to_base64_string (i_blob in blob) return clob;
End fileutils;
/
Show error;
Package body
Set define off;
Create or replace package body fileutils as
Procedure delete_os_file (i_directory varchar2, i_filename varchar2)
is
Begin
utl_file.fremove(i_directory,i_filename);
End;
Procedure move_os_file
(
i_source_directory in varchar2,
i_source_file in varchar2,
i_target_directory in varchar2,
i_target_file in varchar2
)
is
srcdir varchar2(255) := upper(i_source_directory);
tgtdir varchar2(255) := upper(i_target_directory);
begin
--
-- NOTE: If you're getting the all-too-familiar
-- ORA-29292: file rename operation failed
-- and you're SURE that your directory names are correct,
-- and you're SURE that your privileges are correct, both at the
-- OS level, and within the database, there's one last thing that
-- can get you. I learned the hard way that this command will NOT
-- work successfully renaming a file from one filesystem to another,
-- at least when those filesystems are NFS mounted. That is all.
--
utl_file.frename(srcdir,i_source_file,tgtdir,i_target_file,TRUE);
end move_os_file;
Procedure blob_into_file (i_directory in varchar2, i_file_name in varchar2, i_blob in blob)
is
l_file utl_file.file_type;
l_buffer raw(32767);
l_amount binary_integer := 32767;
l_pos integer := 1;
i_blob_len integer;
Begin
i_blob_len := dbms_lob.getlength(i_blob);
l_pos:= 1;
-- Open the destination file.
l_file := utl_file.fopen(i_directory,i_file_name,'wb', 32767);
-- Read chunks of the BLOB and write them to the file
-- until complete.
while l_pos < i_blob_len loop
dbms_lob.read(i_blob, l_amount, l_pos, l_buffer);
utl_file.put_raw(l_file, l_buffer, TRUE);
l_pos := l_pos + l_amount;
end loop;
-- Close the file.
utl_file.fclose(l_file);
End blob_into_file;
Procedure file_into_blob(i_directory in varchar2, i_file_name in varchar2, o_blob out blob)
is
src_loc bfile := bfilename(i_directory, i_file_name);
Begin
-- Initialize the dest blob
o_blob := empty_blob();
-- Open source binary file from OS
dbms_lob.open(src_loc, dbms_lob.lob_readonly);
-- Create temporary LOB object
dbms_lob.createtemporary(
lob_loc => o_blob
, cache => true
, dur => dbms_lob.session
);
-- Open temporary lob
dbms_lob.open(o_blob, dbms_lob.lob_readwrite);
-- Load binary file into temporary LOB
dbms_lob.loadfromfile(
dest_lob => o_blob
, src_lob => src_loc
, amount => dbms_lob.getLength(src_loc));
-- Close lob objects
dbms_lob.close(o_blob);
dbms_lob.close(src_loc);
End file_into_blob;
Function basename (i_filename in varchar2) return varchar2
is
v_basename varchar2(1024);
Begin
--
-- If the regex's below don't match, then it's already at its base name
-- Return what was passed.
--
v_basename := i_filename;
if regexp_like(i_filename,'^.*\\') then
dbms_output.put_line('This is a Windows file');
v_basename := regexp_substr(i_filename,'[^\]*$');
dbms_output.put_line('Basename is : '||v_basename);
end if;
if regexp_like(i_filename,'^/') then
dbms_output.put_line('This is a UNIX file');
v_basename := regexp_substr(i_filename,'[^/]*$');
dbms_output.put_line('Basename is : '||v_basename);
end if;
return v_basename;
End basename;
Function file_exists (i_directory in varchar2, i_filename in varchar2) return boolean
is
v_exists boolean;
v_file_length number;
v_block_size number;
Begin
utl_file.fgetattr(upper(i_directory), i_filename, v_exists, v_file_length, v_block_size);
if (v_exists) then
dbms_output.put_line('File '||i_filename||' exists, '||v_file_length||' bytes');
else
dbms_output.put_line('File '||i_filename||' does not exist');
end if;
return v_exists;
end file_exists;
Procedure convert_clob_to_blob (i_clob in clob, o_blob out blob)
is
v_in pls_Integer := 1;
v_out pls_Integer := 1;
v_lang pls_Integer := 0;
v_warning pls_Integer := 0;
Begin
dbms_lob.createtemporary(o_blob,TRUE);
dbms_lob.converttoblob(o_blob,i_clob,DBMS_lob.getlength(i_clob),v_in,v_out,dbms_lob.default_csid,v_lang,v_warning);
End convert_clob_to_blob;
Procedure convert_blob_to_clob (i_blob in blob, o_clob out clob)
is
v_in pls_Integer := 1;
v_out pls_Integer := 1;
v_lang pls_Integer := 0;
v_warning pls_Integer := 0;
Begin
dbms_lob.createtemporary(o_clob,TRUE);
dbms_lob.converttoclob(o_clob,i_blob,DBMS_lob.getlength(i_blob),v_in,v_out,dbms_lob.default_csid,v_lang,v_warning);
End convert_blob_to_clob;
Procedure blob_to_base64_string (i_blob in blob, o_clob out clob)
is
v_out_cl clob;
file_len pls_integer;
modulo pls_integer;
pieces pls_integer;
amt binary_integer := 23808;
buf raw (32767);
buf_tx varchar2(32767);
pos pls_integer := 1;
filepos pls_integer := 1;
counter pls_integer := 1;
Begin
dbms_lob.createtemporary (v_out_cl, true, dbms_lob.call);
file_len := dbms_lob.getlength (i_blob);
modulo := mod (file_len, amt);
pieces := trunc (file_len / amt);
while (counter <= pieces) loop
dbms_lob.read (i_blob, amt, filepos, buf);
buf_tx:=utl_raw.cast_to_varchar2 (utl_encode.base64_encode (buf));
dbms_lob.writeappend (v_out_cl,length(buf_tx),buf_tx);
filepos := counter * amt + 1;
counter := counter + 1;
end loop;
if (modulo <> 0) THEN
dbms_lob.read (i_blob, modulo, filepos, buf);
buf_tx:=utl_raw.cast_to_varchar2 (utl_encode.base64_encode (buf));
dbms_lob.writeappend (v_out_cl,length(buf_tx),buf_tx);
end if;
o_clob := v_out_cl;
End blob_to_base64_string;
Function blob_to_base64_string (i_blob in blob) return clob
is
v_out_cl clob;
file_len pls_integer;
modulo pls_integer;
pieces pls_integer;
amt binary_integer := 23808;
buf raw (32767);
buf_tx varchar2(32767);
pos pls_integer := 1;
filepos pls_integer := 1;
counter pls_integer := 1;
Begin
dbms_lob.createtemporary (v_out_cl, true, dbms_lob.call);
file_len := dbms_lob.getlength (i_blob);
modulo := mod (file_len, amt);
pieces := trunc (file_len / amt);
while (counter <= pieces) loop
dbms_lob.read (i_blob, amt, filepos, buf);
buf_tx:=utl_raw.cast_to_varchar2 (utl_encode.base64_encode (buf));
dbms_lob.writeappend (v_out_cl,length(buf_tx),buf_tx);
filepos := counter * amt + 1;
counter := counter + 1;
end loop;
if (modulo <> 0) THEN
dbms_lob.read (i_blob, modulo, filepos, buf);
buf_tx:=utl_raw.cast_to_varchar2 (utl_encode.base64_encode (buf));
dbms_lob.writeappend (v_out_cl,length(buf_tx),buf_tx);
end if;
return v_out_cl;
End blob_to_base64_string;
Procedure base64_string_to_blob (i_clob in clob, o_blob out blob)
is
v_out_bl blob;
clob_size number;
pos number;
charBuff varchar2(32767);
dBuffer RAW(32767);
v_readSize_nr number;
v_line_nr number;
begin
dbms_lob.createTemporary (v_out_bl, true, dbms_lob.call);
v_line_nr:=greatest(65, instr(i_clob,chr(10)), instr(i_clob,chr(13)));
v_readSize_nr:= floor(32767/v_line_nr)*v_line_nr;
clob_size := dbms_lob.getLength(i_clob);
pos := 1;
while (pos < clob_size) loop
dbms_lob.read (i_clob, v_readSize_nr, pos, charBuff);
dBuffer := UTL_ENCODE.base64_decode (utl_raw.cast_to_raw(charBuff));
dbms_lob.writeAppend (v_out_bl,utl_raw.length(dBuffer),dBuffer);
pos := pos + v_readSize_nr;
end loop;
o_blob := v_out_bl;
end base64_string_to_blob;
Function base64_string_to_blob (i_clob in clob) return blob
is
v_out_bl blob;
clob_size number;
pos number;
charBuff varchar2(32767);
dBuffer RAW(32767);
v_readSize_nr number;
v_line_nr number;
begin
dbms_lob.createTemporary (v_out_bl, true, dbms_lob.call);
v_line_nr:=greatest(65, instr(i_clob,chr(10)), instr(i_clob,chr(13)));
v_readSize_nr:= floor(32767/v_line_nr)*v_line_nr;
clob_size := dbms_lob.getLength(i_clob);
pos := 1;
while (pos < clob_size) loop
dbms_lob.read (i_clob, v_readSize_nr, pos, charBuff);
dBuffer := UTL_ENCODE.base64_decode (utl_raw.cast_to_raw(charBuff));
dbms_lob.writeAppend (v_out_bl,utl_raw.length(dBuffer),dBuffer);
pos := pos + v_readSize_nr;
end loop;
return v_out_bl;
end base64_string_to_blob;
end fileutils;
/
Show error;
I think you can use LISTAGG function
select listagg(ONE_LINE) WITHIN GROUP (ORDER BY ROW_NUM)
from (
select 'ID,A_ID,B_ID,CREATION_DATE' || CHR(10) as ONE_LINE,
-1 as ROW_NUM
from dual
union all
select ID ||','|| A_ID ||','|| B_ID ||','|| CREATION_DATE || CHR(10) as ONE_LINE,
ROWNUM as ROW_NUM
from Source_Data1
where <expression>
);
The following code will create a CSV version of a table as a CLOB row, using DATA_DUMP.SQL and the function DBMS_XSLPROCESSOR.READ2CLOB.
First, install the DATA_PUMP procedure on your schema. The program is a single command in a single file, you can run it through SQL*Plus or just copy and paste it into your IDE.
Next, create a directory on the server to temporarily store the files. Most existing utilities are meant to create files. It's easier to write a file and read it as a CLOB, than to modify the utility to write directly into a table.
create or replace directory temp_dir as 'C:\temp';
Then this PL/SQL block will read and write the data:
--Store a table as a single CSV clob.
--TODO: Delete the files from the server when done.
declare
v_clob clob;
begin
for tables in
(
--Query that returns the table names.
--(Doesn't have to be hard-coded, maybe a query on DBA_TABLES would help?)
select column_value table_name
from table(sys.odcivarchar2list('source_data'))
) loop
data_dump
(
query_in => 'select * from source_data1',
file_in => 'source_data1.csv',
directory_in => 'temp_dir',
nls_date_fmt_in => 'YYYY-MM-DD HH24:MI:SS',
delimiter_in => ',',
header_row_in => true
);
v_clob := dbms_xslprocessor.read2clob
(
flocation => 'TEMP_DIR',
fname => 'source_data1.csv'
);
insert into result_table (id, source_name, data) values
(result_seq.nextval, 'source_data1', v_clob);
--I wouldn't normally commit after each row, but when exporting large
--tables the script may run too long and there's a good chance of an
--error, so partial results may be helpful.
commit;
end loop;
end;
/
Reading and writing to CSV files is not rocket science but it's not as trivial as most people think. 99% of the CSV programs out there can't handle things like commas in the data, or adding a header row. So it's best to look for preexisting code instead of writing your own utility.
I created a package as follows :
create or replace package apps.xx_sal
as
PROCEDURE SAL_HIST_C(p_person_id IN NUMBER
,p_success OUT VARCHAR2
,p_sql OUT VARCHAR2
,p_cur OUT SYS_REFCURSOR)
IS
l_success VARCHAR2(32000) := 'OK';
l_sql VARCHAR2(32767);
l_sql_empty VARCHAR2(32767);
l_step VARCHAR2(1000);
l_query_length PLS_INTEGER := 0;
c_process CONSTANT VARCHAR2(200) := 'SAL_HIST_C';
BEGIN
l_step := c_process || ' :: BUILD EMPTY SQL';
--
l_sql_empty := ' SELECT person_id
,pay_basis_id
,change_date
from apps.salary_hist_v
WHERE 1 = 2 ';
--
--
l_step := c_process || ' :: BUILD SQL';
--
l_sql :=
'
SELECT paf.person_id
,paf.pay_basis_id
,ppp.change_date change_date
FROM hr.per_pay_proposals ppp
,per_all_assignments_f paf
WHERE paf.primary_flag = ''Y''
AND ppb.pay_basis_id = paf.pay_basis_id
AND paf.person_id = :l_person_id ';
--
-- check sql query
l_query_length := LENGTH(l_sql);
--DBMS_OUTPUT.put_line ('Length of EXECUTED SQL := ' || l_query_length);
-- open cursor with bind variables applied
--
l_step := c_process || ' :: OPEN CURSOR with BIND VARIABLE APPLIED';
--
OPEN p_cur FOR l_sql USING p_person_id;
l_success := 'OK';
p_sql := l_sql;
p_success := l_success;
EXCEPTION
WHEN OTHERS
THEN
l_success := 'ERROR :: ' || l_step || CHR(10) || SQLERRM || CHR(10) || DBMS_UTILITY.format_error_backtrace;
p_sql := l_sql;
p_success := l_success;
OPEN p_cur FOR l_sql_empty;
END SAL_HIST_C;
Now when I am passing the parameter to see the output an error occurs:
DECLARE
--l_person_id number;
l_success VARCHAR2(32000) ;
l_sql VARCHAR2(32767);
l_cur sys_refcursor;
BEGIN
APPS.xx_sal.SAL_HIST_C (
person_id=>4816,
p_success => l_success,
p_sql=>l_sql,
p_cur => :l_cur);
COMMIT;
DBMS_OUTPUT.PUT_LINE ('Output Returned from Proc :: ' || l_success);
DBMS_OUTPUT.PUT_LINE ('SQL Executed for the GRID :: ' || l_cur);
END;
Error:
[Error] Execution (10: 3): ORA-06550: line 10, column 3:
PLS-00306: wrong number or types of arguments in call to 'SAL_HIST_C'
ORA-06550: line 10, column 3:
PL/SQL: Statement ignored
ORA-06550: line 18, column 23:
PLS-00306: wrong number or types of arguments in call to '||'
ORA-06550: line 18, column 1:
PL/SQL: Statement ignored
I don't know why is the error happens. I am executing in Toad. I think I am not calling the procedure in the anonymous block correctly
#user3809240
There are two ORA errors in your message.
1. PLS-00306: wrong number or types of arguments in call to 'SAL_HIST_C'
Replace
APPS.xx_sal.SAL_HIST_C (person_id=>4816, p_success => l_success, p_sql=>l_sql,p_cur => :l_cur);
with
APPS.xx_sal.SAL_HIST_C (4816, l_success, l_sql,l_cur);
2. PLS-00306: wrong number or types of arguments in call to '||'
We cannot do a console output of refcursor. Comment out the below line and then re-run your caller program.
DBMS_OUTPUT.PUT_LINE ('SQL Executed for the GRID :: ' || l_cur);
Now, it should work perfectly.
Cheers,
Madhu.
PS : Please mark this post as ANSWER if my solution works. :)
Two things I can see wrong:
Your stored procedure has a parameter named p_person_id, but you are calling it with a parameter named person_id instead.
You cannot concatenate a string and a ref cursor. If you want to display the values that came out of the ref cursor, you will have to repeatedly fetch values from the cursor, displaying each row, until there aren't any left.
Incidentally I don't see the point of the COMMIT, as there's no data you're inserting, updating or deleting.
I'm in the process of converting some stored procedures from Sybase TSQL to Oracle PL/SQL and I've already come across a problem which I'm struggling to resolve!
The below code will not run:
DECLARE
t INT := 0;
t_error EXCEPTION;
v_line VARCHAR2(100);
BEGIN
SELECT COUNT(*) INTO t FROM user_tables WHERE table_name = UPPER('tbl_BSUK_PriceIssue');
IF t = 1 THEN
EXECUTE IMMEDIATE 'DROP TABLE tbl_BSUK_PriceIssue';
t := 0;
SELECT COUNT(*) INTO t FROM user_tables WHERE table_name = UPPER('tbl_BSUK_PriceIssue');
IF t = 1 THEN
RAISE t_error;
END IF;
END IF;
EXCEPTION
WHEN t_error THEN
v_line := '<<< FAILED DROPPING table tbl_BSUK_PriceIssue >>>';
dbms_output.put_line (v_line);
WHEN OTHERS THEN
v_line := '<<< Unknown Error >>>';
dbms_output.put_line (v_line);
END;
END;
I get the following error message, what am I doing wrong?!
Error starting at line : 17 in command - DECLARE
t INT := 0; t_error EXCEPTION; v_line VARCHAR2(100);
BEGIN
SELECT COUNT(*) INTO t FROM user_tables WHERE table_name =
UPPER('tbl_BSUK_PriceIssue');
IF t = 1 THEN EXECUTE IMMEDIATE 'DROP TABLE tbl_BSUK_PriceIssue';
t := 0; SELECT COUNT(*) INTO t FROM user_tables WHERE table_name =
UPPER('tbl_BSUK_PriceIssue'); IF t = 1 THEN
RAISE t_error; END IF; END IF;
EXCEPTION WHEN t_error THEN v_line := '<<< FAILED DROPPING table
tbl_BSUK_PriceIssue >>>'; dbms_output.put_line (v_line); WHEN
OTHERS THEN
v_line := '<<< Unknown Error >>>';
dbms_output.put_line (v_line); END;
END; Error report - ORA-06550: line 30, column 1: PLS-00103: Encountered the symbol "END"
06550. 00000 - "line %s, column %s:\n%s"
*Cause: Usually a PL/SQL compilation error.
*Action:
I'm actually trying to replace the following TSQL with a PL/SQL version:
-- Create temp table for relevant trev_id's
IF OBJECT_ID('dbo.tbl_BSUK_PriceIssue') IS NOT NULL
BEGIN
DROP TABLE dbo.tbl_BSUK_PriceIssue
IF OBJECT_ID('dbo.tbl_BSUK_PriceIssue') IS NOT NULL
PRINT '<<< FAILED DROPPING TABLE dbo.tbl_BSUK_PriceIssue >>>'
ELSE
PRINT '<<< DROPPED TABLE dbo.tbl_BSUK_PriceIssue >>>'
END
go
try to remove END; in this section
WHEN OTHERS THEN
v_line := '<<< Unknown Error >>>';
dbms_output.put_line (v_line);
END;
UPD. Actually, you can do it a bit shorter, no need to check if table exists after drop
declare
eTableNotExists exception;
pragma exception_init(eTableNotExists, -00942);
begin
EXECUTE IMMEDIATE 'DROP TABLE tbl_BSUK_PriceIssue';
dbms_output.put_line('<<< DROPPED TABLE dbo.tbl_BSUK_PriceIssue >>>');
exception
when eTableNotExists then null
when others then
dbms_output.put_line ('<<< Unknown Error >>>' || sqlerrm);
end;
/
I don't know about the error, but you can do what you want in a fraction of the code. You don't need the count variable if you use EXISTS() and you don't need EXECUTE IMMEDIATE because you don't have variable data in the command:
IF 1 = (SELECT 1 FROM user_tables WHERE table_name = 'TBL_BSUK_PRICEISSUE') THEN
DROP TABLE tbl_BSUK_PriceIssue;
IF 1 = (SELECT 1 FROM user_tables WHERE table_name = 'TBL_BSUK_PRICEISSUE') THEN
RAISE EXCEPTION;
END IF;
END IF;