How to load csv files into a table - sql

Good Morning, I've been trying for some days to load data from a csv file into a table, BUT I found out how to do it if you have privileges to create a new directory objects.
I did it locally with both ways UTL_FILE and external tables, but, I need special privileges to do it.
But when I tried to do it online I found out that I haven't got those privileges so, I'd like to find out if there's any way to do it without them.
Thank you!
PS: The two approaches are as follows:
CREATE TABLE TEST
(
A varchar2(25);
B number;
)
WITH UTL_FILE:
PROCEDURE load AS
F UTL_FILE.FILE_TYPE;
V_LINE VARCHAR2(4000);
v_A varchar2(25);
v_B number;
BEGIN
F := UTL_FILE.FOPEN (<directory>, '<csv file>', 'r');
IF UTL_FILE.IS_OPEN(F) THEN
LOOP
BEGIN
UTL_FILE.GET_LINE(F, V_LINE, 1000);
IF V_LINE IS NULL THEN
EXIT;
END IF;
v_A := REGEXP_SUBSTR(V_LINE, '[^;]+', 1, 1);
v_B := REGEXP_SUBSTR(V_LINE, '[^;]+', 1, 2);
INSERT INTO test(v_A, v_B);
COMMIT;
EXCEPTION
WHEN NO_DATA_FOUND THEN
EXIT;
END;
END LOOP;
END IF;
UTL_FILE.FCLOSE(F);
END load;
WITH external tables
PROCEDURE loadcsv AS
A varchar2(25);
B number;
BEGIN
INSERT INTO test select * testload;
COMMIT;
END loadcsv;
HAVING
CREATE TABLE testload
(
A VARCHAR2(25)
B number
)
ORGANIZATION EXTERNAL
( TYPE ORACLE_LOADER
DEFAULT DIRECTORY "PRUEBAS"
ACCESS PARAMETERS
( records delimited by newline
fields terminated by ';'
)
LOCATION
( 'Log_CSV_REND_SICTG_20170507.csv'
)
)
REJECT LIMIT UNLIMITED ;

You might want to try loading with SQL Loader. It is easy to use and very fast. You have to create a control file that describes how the data will be loaded, and needs to have the same structure as the table you're trying to load into. For you case it might look something like this.
test.ctl
LOAD DATA
INFILE <yourfilepath>
APPEND INTO TABLE TEST
FIELDS TERMINATED BY ','
(
A,
B
)
Then, from the command line.
host sqlldr username/password control=youfolder\test.ctl
log=yourfolder\test.log

Related

performance issue when inserting large records

I am parsing string into comma separated and inserting them to global table. The performance is good when inserting around 5k records, performance sucks if the inserting record is around 40k+. The global table has only one column. I thought using bulk fetch and forall will increase the performance, but it’s not the case so far. How can I rewrite below insertion query or any other ways this can be achieved for inserting large records? help will be highly appreciated. I did testing by running insert query by its own and it’s taking long time to process if data size is large.
//large string
emp_refno in CLOB;
CREATE OR replace PROCEDURE employee( emp_refno IN CLOB ) AS
c_limit PLS_INTEGER := 1000;
CURSOR token_cur IS
WITH inputs(str) AS
( SELECT to_clob(emp_refno)
FROM dual ),
prep(s,n,token,st_pos,end_pos ) AS
(
SELECT ','|| str || ',',-1,NULL,NULL,1
FROM inputs
UNION ALL
SELECT s, n + 1,substr(s, st_pos, end_pos - st_pos),
end_pos + 1,instr(s, ',', 1, n + 3)
FROM prep
WHERE end_pos != 0
)
SELECT token
FROM prep
WHERE n > 0;
TYPE token_t
IS
TABLE OF CLOB;
rec_token_t TOKEN_T;
BEGIN
OPEN token_cur;
LOOP
FETCH token_cur bulk collect
INTO rec_token_t limit c_limit;
IF rec_token_t.count > 0 THEN
forall rec IN rec_token_t.first ..rec_token_t.last
INSERT INTO globaltemp_emp
VALUES ( rec_token_t(rec) );
COMMIT;
END IF;
EXIT
WHEN rec_token_t.count = 0;
END LOOP;
OPEN p_resultset FOR
SELECT e.empname,
e.empaddress,
f.department
FROM employee e
join department f
ON e.emp_id = t.emp_id
AND e.emp_refno IN
(
SELECT emp_refno
FROM globaltemp_emp) //USING gtt IN subquery
END;
I have adapted a function which gives better performance.For 90k records, it returns in 13 seconds.Also reduce the c_limit to 250
You can adapt the below
CREATE OR replace FUNCTION pipe_clob ( p_clob IN CLOB,
p_max_lengthb IN INTEGER DEFAULT 4000,
p_rec_delim IN VARCHAR2 DEFAULT '
' )
RETURN sys.odcivarchar2list pipelined authid current_user AS
/*
Break CLOB into VARCHAR2 sized bites.
Reduce p_max_lengthb if you need to expand the VARCHAR2
in later processing.
Last record delimiter in each bite is not returned,
but if it is a newline and the output is spooled
the newline will come back in the spooled output.
Note: this cannot work if the CLOB contains more than
<p_max_lengthb> consecutive bytes without a record delimiter.
*/
l_amount INTEGER;
l_offset INTEGER;
l_buffer VARCHAR2(32767 byte);
l_out VARCHAR2(32767 byte);
l_buff_lengthb INTEGER;
l_occurence INTEGER;
l_rec_delim_length INTEGER := length(p_rec_delim);
l_max_length INTEGER;
l_prev_length INTEGER;
BEGIN
IF p_max_lengthb > 4000 THEN
raise_application_error(-20001, 'Maximum record length (p_max_lengthb) cannot be greater than 4000.');
ELSIF p_max_lengthb < 10 THEN
raise_application_error(-20002, 'Maximum record length (p_max_lengthb) cannot be less than 10.');
END IF;
IF p_rec_delim IS NULL THEN
raise_application_error(-20003, 'Record delimiter (p_rec_delim) cannot be null.');
END IF;
/* This version is limited to 4000 byte output, so I can afford to ask for 4001
in case the record is exactly 4000 bytes long.
*/
l_max_length:=dbms_lob.instr(p_clob,p_rec_delim,1,1)-1;
l_prev_length:=0;
l_amount := l_max_length + l_rec_delim_length;
l_offset := 1;
WHILE (l_amount = l_max_length + l_rec_delim_length
AND
l_amount > 0)
LOOP
BEGIN
dbms_lob.READ ( p_clob, l_amount, l_offset, l_buffer );
EXCEPTION
WHEN no_data_found THEN
l_amount := 0;
END;
IF l_amount = 0 THEN
EXIT;
ELSIF lengthb(l_buffer) <= l_max_length THEN
pipe ROW(rtrim(l_buffer, p_rec_delim));
EXIT;
END IF;
l_buff_lengthb := l_max_length + l_rec_delim_length;
l_occurence := 0;
WHILE l_buff_lengthb > l_max_length
LOOP
l_occurence := l_occurence + 1;
l_buff_lengthb := instrb(l_buffer,p_rec_delim, -1, l_occurence) - 1;
END LOOP;
IF l_buff_lengthb < 0 THEN
IF l_amount = l_max_length + l_rec_delim_length THEN
raise_application_error( -20004, 'Input clob at offset '
||l_offset
||' for lengthb '
||l_max_length
||' has no record delimiter' );
END IF;
END IF;
l_out := substrb(l_buffer, 1, l_buff_lengthb);
pipe ROW(l_out);
l_prev_length:=dbms_lob.instr(p_clob,p_rec_delim,l_offset,1)-1;--san temp
l_offset := l_offset + nvl(length(l_out),0) + l_rec_delim_length;
l_max_length:=dbms_lob.instr(p_clob,p_rec_delim,l_offset,1)-1;--san temp
l_max_length:=l_max_length-l_prev_length;
l_amount := l_max_length + l_rec_delim_length;
END LOOP;
RETURN;
END;
and then use like the below in the cursor in your procedure
CURSOR token_cur IS
select * from table (pipe_clob(emp_refno||',',10,','));
Three quick suggestions:
Perform commit for around 1000(or in batches) records rather than doing for each.
Replace in with exists for the Ref cursor.
Index globaltemp_emp.emp_refno if it doesn't have already.
Additionally recommend to run explain plan for each of the DML operation to check for any odd behaviour.
user uploads text file and I parse that text file as a comma seperated string and pass it to Oracle DB.
You are doing a bunch of work to turn that file into a string and then another bunch of work to convert that string into a table. As many people have observed before me, the best performance comes from not doing work we don't have to do.
In this case this means you should load the file's contents directly into the database. We can do this with an external table. This is a mechanism which allows us to query data from a file on the server using SQL. It would look something like this:
create table emp_refno_load
(emp_refno varchar2(24))
organization external
(type oracle_loader
default directory file_upload_dir
access parameters
(records delimited by newline
fields (employee_number char(24)
)
)
location ('some_file.txt')
);
Then you can discard your stored procedure and temporary table and re-write your query to something like this:
SELECT e.empname,
e.empaddress,
f.department
FROM emp_refno_load l
join employee e ON l.emp_refno = e.emp_refno
join department f ON e.emp_id = f.emp_id
The one snag with external tables is they require access to an OS directory (file_upload_dir in my example above) and some database security policies are weird about that. However the performance benefits and simplicity of approach should carry the day.
Find out more.
An external table is undoubtedly the most performative approach (until you hit millions of roads and then you need SQL*Loader ).

how to compare two array data and listing out missing data -- collections exists method

I have requirement as below, have two list, find out which element is missing from list comparing with other list. In below example used exist method, but after executing this I came to know we cannot use exist in this way, it will work only with index. please guide me which method will help me to meet this type of requirement
TABLE - MISSEVETEST(PID, DIR, EVE_NAME)
SAMPLE DATA - 111, O, CN.. 111, O, SN output:- is RPI is missing
CREATE OR REPLACE PROCEDURE EVE_MISS_POC IS
TYPE t_missing_eve IS TABLE OF MISSEVETEST%ROWTYPE;
list_missing_eve t_missing_eve := t_missing_eve();
TYPE t_out_eve IS VARRAY(3) OF VARCHAR2(80);
varray_out_eve t_out_eve;
CURSOR c_missing_eve is SELECT PID, DIR, EVE_NAME FROM MISSEVETEST;
current_pid PAY.PID%TYPE;
TYPE t_pid_present_eve IS VARRAY(5) OF VARCHAR2(80);
varray_pid_present_eve t_puid_present_eve := t_puid_present_eve();
BEGIN
DBMS_OUTPUT.PUT_LINE('PROCESS STARTRED');
varray_out_eve := t_out_eve('CN', 'SN', 'RPI');
OPEN c_missing_eve;
LOOP
-- processing payment records in batches. Each batch - 10000 records
FETCH c_missing_eve bulk collect INTO list_missing_eve limit 10000;
BEGIN
current_pid := list_missing_eve(1).PID; --assign first pid from list
FOR index_eve IN 1 .. list_missing_eve.count
LOOP
--this will collect all events belongs to same puid
IF current_pid = list_missing_eve(index_eve).PID THEN
varray_pid_present_eve.EXTEND;
-- collect all events of same puid
varray_pid_present_eve(varray_pid_present_eve.count) := list_missing_eve(index_eve).EVE_NAME;
ELSE
FOR i IN 1..varray_out_eve .COUNT LOOP
IF varray_pid_present_eve.EXISTS(varray_out_eve (i)) THEN
DBMS_OUTPUT.PUT_LINE('EVENT:' ||varray_out_eve (i) ||' EXIST');
ELSE
DBMS_OUTPUT.PUT_LINE('EVENT:' ||varray_out_eve (i) ||' NOT EXIST');
END IF;
END LOOP;
END IF;
END LOOP;
END;
EXIT WHEN c_missing_eve%notfound;
END LOOP; -- End for limit loop
CLOSE c_missing_eve;
END;
If you had used NESTED TABLE instead of VARRAY, you could have used the MULTISET EXCEPT operator.
SET SERVEROUTPUT ON
DECLARE
TYPE vtype IS TABLE OF VARCHAR2 (10);
v1 vtype
:= vtype ('ABC',
'DEF',
'GHI',
'KLM');
v2 vtype := vtype ('ABC', 'DEF');
v3 vtype := vtype ();
BEGIN
v3 := v1 MULTISET EXCEPT v2;
FOR i IN v3.FIRST .. v3.LAST
LOOP
DBMS_OUTPUT.PUT_LINE (v3 (i));
END LOOP;
END;
/
GHI
KLM
PL/SQL procedure successfully completed.

How to update CLOB column from a physical file?

I have a CLOB column in a table which holds very large amount of XML data. I need to update this column's value for one row of table. How can I do this?
I have tried googling, but this visibly simple stuff is not available in a simple language anywhere. Can anybody please suggest?
If I use the normal update query syntax and paste the huge xml content inside the single quotes (the single quote in xml content replaced with 2 single quotes), then the sql developer just disables the execute query button.
update tableName t
set t.clobField = 'How to specify physical file data'
where t.anotherField='value';
You need to create a function that reads the data from the file into a variable of the CLOB data type and returns it as the result.
Try this working example:
create table tclob (id number, filename varchar2 (64), doc clob)
/
insert into tclob values (1, 'test.xml', empty_clob ());
commit;
create or replace function clobLoader (filename varchar2) return clob is
bf bfile := bfilename ('TEMPFILES', filename);
cl clob;
begin
if dbms_lob.fileexists (bf) = 1 then
dbms_lob.createtemporary (cl, true);
dbms_lob.fileopen (bf, dbms_lob.file_readonly);
dbms_lob.loadfromfile (cl, bf, dbms_lob.getlength (bf));
dbms_lob.fileclose (bf);
else cl := empty_clob ();
end if;
return cl;
end;
/
Usage:
update tclob t
set t.doc = clobLoader (t.filename)
where t.id = 1;
1 row updated.
Search the internet for "load clob from file" and you will find a couple of examples. Such as this (http://www.anujparashar.com/blog/loading-text-file-into-clob-field-in-oracle):
DECLARE
v_bfile BFILE;
v_clob CLOB;
BEGIN
v_bfile := BFILENAME (p_file_directory, p_file_name);
IF DBMS_LOB.FILEEXISTS (v_bfile) = 1 THEN
DBMS_LOB.OPEN (v_bfile);
DBMS_LOB.CREATETEMPORARY (v_clob, TRUE, DBMS_LOB.SESSION);
DBMS_LOB.LOADFROMFILE (v_clob, v_bfile, DBMS_LOB.GETLENGTH (v_bfile));
DBMS_LOB.CLOSE (v_bfile);
INSERT INTO tbl_clob (clob_col)
VALUES (v_clob);
END IF;
COMMIT;
END;
/

Read text file to insert data into Oracle SQL table

I am studying Oracle SQL developer.
What I am doing is reading text file line by line from the folder.
Then Inserting data to the SQL table.
I am able to compile my PROCEDURE, however, it doesn't seem to be inserting the data to file.
Create or Replace PROCEDURE Rfile is
f UTL_FILE.FILE_TYPE;
s VARCHAR2(200);
BEGIN
f := UTL_FILE.FOPEN('C:\Projects\','testdatabinary.txt','R');
IF UTL_FILE.IS_OPEN(f) THEN
LOOP
BEGIN
UTL_FILE.GET_LINE(f,s);
IF s IS NULL THEN
EXIT;
END IF;
INSERT INTO DATAINSERT
(COLUMN1, COLUMN2)
VALUES
(s, 'testdatabinary');
END;
END LOOP;
COMMIT;
END IF;
END;
And I have a table DATAINSERT with two varchar(200) type cols
I am not really sure the reasons that PROCEDURE is not inserting data to table
I just checked error message
Error starting at line 1 in command:
EXEC Rfile
ORA-29280: invalid directory path
ORA-06512: at "SYS.UTL_FILE", line 41
ORA-06512: at "SYS.UTL_FILE", line 478
ORA-06512: at "SYSTEM.RFILE", line 5
Not sure what causing problems. For me its working fine here is my example code
--Reference Site
--https://community.oracle.com/thread/3633577?start=0&tstart=0
set serveroutput on;
CREATE or replace DIRECTORY USER_DIR AS '/home/oracle';
GRANT READ ON DIRECTORY USER_DIR TO PUBLIC;
DECLARE
V1 VARCHAR2(200); --32767
F1 UTL_FILE.FILE_TYPE;
BEGIN
F1 := UTL_FILE.FOPEN('USER_DIR','temp.txt','R');
Loop
BEGIN
UTL_FILE.GET_LINE(F1,V1);
dbms_output.put_line(V1);
EXCEPTION WHEN No_Data_Found THEN EXIT; END;
end loop;
IF UTL_FILE.IS_OPEN(F1) THEN
dbms_output.put_line('File is Open');
end if;
UTL_FILE.FCLOSE(F1);
END;
/
set serveroutput off;
You can't able to enter the path directly to the file open command like this
f := UTL_FILE.FOPEN('C:\Projects\','testdatabinary.txt','R');
Instead of entering path directly, you have to create directory using the below sql query
CREATE or replace DIRECTORY USER_DIR AS 'C:\PROJECTS\';
Then enter the directory name to the file open command. It will be like this
f := UTL_FILE.FOPEN('USER_DIR ','testdatabinary.txt','R');
First procedure trying to load the file from PC into the server whereas in second procedure file is from server to server. UTL_FILE.FOPEN works for server-side PL/SQL. So first procedure wont execute properly... If you want to load a file from PC to server, then you need to user any front end like D#K or VB.
The mistake your making here is using the workstation path, you actually need to be using the directory path.
The quickest way to resolve this, is just to use the WINSCP command.
Use WINSCP to tell you what the directory path is, then simply substitute this new information for the old path you have in your code, and everything should work ok.
This is very conceptual and tricky. which you can find more concepts and details below:
https://ora-data.blogspot.in/2016/11/read-and-writ-text-file-using-UTLFILE.html
Sure will be helpful:
create or replace procedure read_file is
f_line varchar2(2000);
f utl_file.file_type;
f_dir varchar2(250);
fname varchar2(50);
Comma1 varchar(10);
Comma2 varchar(10);
Comma3 varchar(10);
Comma4 varchar(10);
Comma5 varchar(10);
f_empno emp.empno%type;
f_ename emp.ename%type;
f_job emp.job%type;
f_mgr emp.mgr%type;
f_hiredate emp.hiredate%type;
f_sal emp.sal%type;
begin
f_dir := ‘E:\PLSQL’;
fname := ‘input.txt’;
f := utl_file.fopen(‘UTL_FILE_DIR’,fname,’r’); –opening the file using fopen function
loop
begin
utl_file.get_line(f,f_line);
–using a loop continuously get the file’s content using get_line function exception when no_data_found then
exit;
end;
Comma1 := INSTR(f_line, ‘,’ ,1 , 1);
Comma2 := INSTR(f_line, ‘,’ ,1 , 2);
Comma3 := INSTR(f_line, ‘,’ ,1 , 3);
Comma4 := INSTR(f_line, ‘,’ ,1 , 4);
Comma5 := INSTR(f_line, ‘,’ ,1 , 5);
–Each field in the input record is delimited by commas.
–We need to find the location of two commas in the line.
–and use the locations to get the field from the line.
f_empno := to_number(SUBSTR(f_line, 1, Comma1-1));
f_ename := SUBSTR(f_line, Comma1+1, Comma2-Comma1-1);
f_job := SUBSTR(f_line, comma2+1, Comma3-Comma2-1);
f_mgr := to_number(SUBSTR(f_line, comma3+1, Comma4-Comma3-1));
f_hiredate := to_date(SUBSTR(f_line, comma4+1, Comma5-Comma4-1),’dd-mon-yyyy’);
f_sal := to_number(SUBSTR(f_line, comma5+1),’99999′); dbms_output.put_line(f_empno ||’ ‘|| f_ename || ‘ ‘ || f_job || ‘ ‘ || f_mgr ||’ ‘ || f_hiredate||’ ‘|| f_sal);
insert into emp12 VALUES (f_empno,f_ename,f_job,f_mgr,f_hiredate,f_sal);
end loop;
utl_file.fclose(f);
commit;
end;
/
first login with
username:sys as sysdba
password should be the same as used for user 'system'
now enter
SQL> grant execute on UTL_FILE to PUBLIC;
Now login with any user with which you want to create procedure

Error ORA-06502 in BLOB retrieval

I'm trying to develop a procedure which will read a BLOB field of a table and write an RTF document which will be used as template to another procedure but when the procedure fires up the select statemente, i got stuck with a ORA-06502 error.
This error, after reading the documentation, is caused by incompatibilities between fields (numeric or value error string)
But i've seen this example everywhere in the internet and i'm running out of ideas of what is causing it.
The source code of my procedure follows:
PROCEDURE p_transfer_db_client(pcPath IN VARCHAR2,
pnSequence IN NUMBER) IS
v_src_blob BLOB;
v_file UTL_FILE.FILE_TYPE;
v_offset INTEGER := 1;
v_amount BINARY_INTEGER := 32766;
v_binary_buffer RAW(32767);
BEGIN
SELECT model
INTO v_src_blob
FROM models
wHERE id = pnSequence;
v_file := UTL_FILE.FOPEN(pcPath, 'model.rtf', 'wb', v_amount);
LOOP
BEGIN
DBMS_LOB.READ(v_src_blob, v_amount, v_offset, v_binary_buffer);
UTL_FILE.PUT_RAW(v_file, v_binary_buffer);
v_offset := v_offset + v_amount;
EXCEPTION
WHEN NO_DATA_FOUND THEN
EXIT;
END;
END LOOP;
UTL_FILE.FFLUSH(v_file);
UTL_FILE.FCLOSE(v_file);
END p_transfer_db_client;
Edit: I hadn't seen that you had tagged this question as forms.
Your code won't work inside forms: UTL_FILE and DBMS_LOB run on the DB server, not the forms client so they should be in a PL/SQL procedure, not a Forms procedure.
I'm pretty sure the BLOB datatype is not managed like this in forms, in fact I'm surprised it even compiles!
Take a look at this example showing how to handle blobs inside forms for instance, and you will see that you need to define and query the blob inside a PLSQL procedure, not directly through forms.
I suggest you create the procedure inside the DB as a PLSQL package and call this package from your forms client.
There is something else going on, I can't see why the SELECT statement wouldn't work.
Do you still get the error if you comment out all variables declarations and all the following statements, i.e:
SQL> create table models (
2 inst_num_instituicao NUMBER(2) not null,
3 id NUMBER(3) not null,
4 model BLOB not null);
Table created.
SQL> insert into models values (1, 1, hextoraw('FFFF'));
1 row created.
SQL> declare
2 b blob;
3 begin
4 select model into b from models where id = 1;
5 end;
6 /
PL/SQL procedure successfully completed.