I'm writing a programm in PL/SQL for downloading XML files from internet, parsing them and storing values in Oracle database. I have one large XML file, which includes links for a huge amount of smaller XMLs. There are about 6253 files to be parsed. I have function, which downloads XML file into CLOB and saves data into XmlType. This value is returned into programm and is further processed. This is the function:
create or replace function get_xml_by_url
( v_url VARCHAR2
)
RETURN XMLType
AS
req SYS.UTL_HTTP.REQ;
resp SYS.UTL_HTTP.RESP;
xmlClob CLOB;
x XmlType;
l_offset number := 1;
value VARCHAR2(3999); -- URL to post to
BEGIN
BEGIN
UTL_HTTP.SET_PROXY('http://10.1.250.233:8080');
req := UTL_HTTP.BEGIN_REQUEST (url=> v_url, method => 'GET');
UTL_HTTP.SET_HEADER(req, 'User-Agent', 'Mozilla/4.0');
UTL_HTTP.SET_HEADER
( r => req
, name => 'Content-Type'
, value => 'text/xml;charset=UTF-8'
);
resp := UTL_HTTP.GET_RESPONSE(req);
DBMS_LOB.CREATETEMPORARY(xmlClob, true);
-- Loading first line
UTL_HTTP.READ_LINE(resp,value,false);
DBMS_LOB.WRITE(xmlClob,length(value),l_offset,value);
l_offset := l_offset + length(value);
-- Loading and adjusting second line
UTL_HTTP.READ_LINE(resp,value,true);
value := rtrim(value,'xmlns="http://seznam.gov.cz/ovm/datafile/seznamovm/v1">')||'>';
DBMS_LOB.WRITE(xmlClob, length(value), l_offset,value);
l_offset := l_offset + length(value);
-- Filling CLOB
LOOP
UTL_HTTP.READ_LINE(resp,value,false);
DBMS_LOB.WRITE(xmlClob,length(value),l_offset,value);
l_offset := l_offset + length(value);
END LOOP;
EXCEPTION
when UTL_HTTP.END_OF_BODY
then
UTL_HTTP.END_RESPONSE(resp);
when others
then
utl_http.end_response(resp);
END;
x := XMLType.createXML(xmlClob);
DBMS_LOB.FREETEMPORARY(xmlClob);
RETURN x;
END;
I was calling this function in a loop for all 6.253 XMl files and every time I've got an error, but every time in different file, and when I ran the script again for only the one XML file which raised an error, it ran fine. I think that the problem is something about memory, but I don't know where a nd why it occurs.
I'm getting following error:
Error report:
ORA-31011: XML parsing failed
ORA-19202: Error occurred in XML processing
LPX-00249: invalid external ID declaration
Error at line 1
ORA-06512: in "SYS.XMLTYPE", line 5
ORA-06512: in "GET_XML_BY_URL", line 47
ORA-06512: in line 29
31011. 00000 - "XML parsing failed"
*Cause: XML parser returned an error while trying to parse the document.
*Action: Check if the document to be parsed is valid.
GET_XML_BY_URL is a name given to described function. Does anybody have any experience wuth this kind of problem?
Best regards, Michal
Related
How can I pass a variable value inside the xpath of xmltable?
DECLARE
v NUMBER := 0;
BEGIN
SELECT *
FROM xml_billrun_files t ,
xmltable('/invoice/AR_ITEMS[#elem='||v||']/ITEMS/USAGE_RECORDS/SESSION_INFO'
passing t.update_xmldoc
columns chrg_duration VARCHAR2(20) path 'DURATION',
amount NUMBER path 'AMOUNT') x;
END;
I've tried this one.
DECLARE
v NUMBER := 0;
BEGIN
SELECT *
FROM xml_billrun_files t ,
xmltable('/invoice/AR_ITEMS[#elem=$i]/ITEMS/USAGE_RECORDS/SESSION_INFO'
passing t.update_xmldoc, xmltype(v) as "i"
columns chrg_duration VARCHAR2(20) path 'DURATION',
amount NUMBER path 'AMOUNT') x;
END;
But it returns an error:
ORA-31011: XML parsing failed
ORA-19202: Error occurred in XML processing
LPX-00210: expected '<' instead of '0'
Error at line 1
ORA-06512: at "SYS.XMLTYPE", line 310
ORA-06512: at line 1
ORA-06512: at line 5
31011. 00000 - "XML parsing failed"
*Cause: XML parser returned an error while trying to parse the document.
*Action: Check if the document to be parsed is valid.
thanks for the help.
In my answer to your previous question, I incorrectly used xmltype(lp) as "lp" for your XMLQuery call. Not sure why it didn't complain there, but it didn't actually restrict the match anyway...
For this XMLTable call you can pass the number directly, without conversion/cast, as it's already a number:
SELECT chrg_duration, amount
into ...
FROM xml_billrun_files t ,
xmltable('/invoice/AR_ITEMS[#elem=$i]/ITEMS/USAGE_RECORDS/SESSION_INFO'
passing t.update_xmldoc, v as "i"
columns chrg_duration VARCHAR2(20) path 'DURATION',
amount NUMBER path 'AMOUNT') x;
If you're doing it in a for loop, then the index is the wrong data type, and you need to cast it to number:
SELECT chrg_duration, amount
into ...
FROM xml_billrun_files t ,
xmltable('/invoice/AR_ITEMS[#elem=$i]/ITEMS/USAGE_RECORDS/SESSION_INFO'
passing t.update_xmldoc, cast(v as number) as "i"
columns chrg_duration VARCHAR2(20) path 'DURATION',
amount NUMBER path 'AMOUNT') x;
I have package with Period and TableOfPeriod types:
TYPE Period
IS RECORD
( StartPeriod Date,
EndPeriod Date
);
TYPE TableOfPeriod
IS TABLE OF Period;
and in this package I have three simple function:
FUNCTION Temp1
RETURN TableOfPeriod IS
returnedValue TableOfPeriod := TableOfPeriod();
BEGIN
returnedValue.extend(1);
returnedValue(1).StartPeriod := sysdate-100;
returnedValue(1).EndPeriod := sysdate;
RETURN returnedValue;
END Temp1;
FUNCTION CalculateFine
return VARCHAR2
IS
freewillLockTableRP TableOfPeriod:=TableOfPeriod();
compulsoryLockTableRP TableOfPeriod:=TableOfPeriod();
BEGIN
--for testing
compulsoryLockTableRP:=Temp1();
FOR i IN compulsoryLockTableRP.FIRST..compulsoryLockTableRP.LAST LOOP
IF(((compulsoryLockTableRP(i).EndPeriod - compulsoryLockTableRP(i).StartPeriod)>1)) THEN
BEGIN
-- RAISE_APPLICATION_ERROR(-20001, 'Hello world');
SELECT T111.StartPeriod StartPeriod,
T111.EndPeriod EndPeriod
bulk collect into freewillLockTableFull
FROM TABLE(DistributeDaysByPeriods(compulsoryLockTableRP, 5)) T111;
END;
END IF;
END LOOP;
/*SELECT T111.StartPeriod StartPeriod,
T111.EndPeriod EndPeriod
-- BULK COLLECT INTO compulsoryLockTableRP
bulk collect into freewillLockTableFull
FROM TABLE(DistributeDaysByPeriods(compulsoryLockTableRP, 5)) T111;*/
--===========
--SOME OTHER PROCESSING
RETURN 'Ok '
|| '#r';
EXCEPTION
WHEN No_Data_Found THEN return 'No data found#g';
-- WHEN OTHERS THEN RETURN SQLERRM;
END CalculateFine;
When I execute this function, I have next error:
"ORA-21700: object does not exist or is marked for delete ORA-06512:
at "MyPackageName", line 1181 ORA-06512: at line 1
21700. 00000 - "object does not exist or is marked for delete""
where 1181 line is a line with Select statement of CalculateFine function. Can anybody tell me, whats wrong and how I can solve this problem?
Check if you have in the same folder where your package is located a file with the same name but with an extention *.~sql (e.g.: your_file_name.~sql). I had the same error but after I have deleted an *.~sql file I could compile my package without ORA errors.
i m seeing some strange behavior for my pdf to text conversion using oracle
bellow is the code of a sql file.
create or replace directory pdf_dir as '&1';
create or replace directory l_curr_dir as '&3';
declare
ll_clob CLOB;
l_bfile BFILE;
l_filename VARCHAR2(100) := '&2';
begin
begin
ctx_ddl.drop_policy('test_policy');
exception
when others then
null;
end;
ctx_ddl.create_policy('test_policy','ctxsys.auto_filter');
l_bfile := bfilename('PDF_DIR',l_filename);
dbms_lob.createtemporary(ll_clob, true);
ctx_doc.policy_filter(
policy_name => 'test_policy'
, document => l_bfile
, restab => ll_clob
, plaintext => true
);
ll_clob := REPLACE(TRIM(ll_clob), chr(13), chr(10));
ll_clob := REPLACE(ll_clob, chr(10), chr(32) || '<<EOL>>' || chr(10)||'<<BOL>>');
INSERT into tempclob_op(filename, data) VALUES(l_filename, ll_clob);
DBMS_XSLPROCESSOR.clob2file (ll_clob,'L_CURR_DIR' , 'plaintext.text');
dbms_lob.freeTemporary( ll_clob );
end;
/
problem is i have run this code for 10000 files and it gives correct results for almost all but for almost 10 files it corrupts the output in plaintext.text file. And i dont know why is it happening? Also when i run this sql code for individual files it gives me correct results.
I have added some delay of 2 seconds in every execution while in loop for each file. and seems it resolved the problem strangely ..no concrete answers though.
I have created a Store Procedure to generate an xml file of Table Data,
but in my database some table have "extended ASCII codes (character code 128-255)".
When I generate xml file then it shows error "ORA-31061: XDB error: special char to escaped char conversion failed."
So I replaced these Char to space but I need all ASCII codes (character code 128-255) in XML files.
Please help
My Store Procedure is below:
create or replace
PROCEDURE Export_project6
(
V_TABLE_NAME1 IN varchar2,
v_FLAG OUT NUMBER
)
AS
BEGIN
----- Export table data
DECLARE
v_file UTL_FILE.file_type;
qryCtx DBMS_XMLGEN.ctxHandle;
result CLOB;
v_FILENAME varchar2(50);
V_TABLE_NAME varchar2(50);
xt_data xmltype;
v_ctx dbms_xmlgen.ctxHandle;
rc_data sys_refcursor;
BEGIN
V_TABLE_NAME := UPPER(V_TABLE_NAME1) ;
v_file := UTL_FILE.fopen('MYXML',V_TABLE_NAME||'.xml', 'W');
OPEN rc_data FOR
'select * FROM '||V_TABLE_NAME||' ORDER BY 1' ;
v_ctx := dbms_xmlgen.newContext (rc_data);
DBMS_XMLGEN.USEITEMTAGSFORCOLL (v_ctx);
DBMS_XMLGEN.SETNULLHANDLING(v_ctx, 1);
DBMS_XMLGEN.setrowsettag(v_ctx,'root');
DBMS_XMLGEN.setrowtag(v_ctx,V_TABLE_NAME );
result:= DBMS_XMLGEN.getXML(v_ctx);
result := REPLACE( result, '<?xml version="1.0"?>','<?xml version="1.0" encoding="UTF-8" standalone ="yes"?>');
-- DBMS_XMLGEN.RESTARTQUERY (v_ctx);
-- xt_data := dbms_xmlgen.getXMLType (v_ctx);
dbms_xslprocessor.clob2file( result, 'MYXML', ''||V_TABLE_NAME||'.xml',1);
dbms_xmlgen.closeContext (v_ctx);
v_FLAG := 1;
EXCEPTION
WHEN OTHERS THEN
DBMS_OUTPUT.PUT_LINE(SQLERRM);
DBMS_XMLGEN.closeContext (v_ctx);
v_FLAG := 0;
END ;
Please Advices and help here
END Export_project6;
At least part of the problem is that you're specifying the file should be US7ASCII, which only allows the first 128 ASCII characters, not the extended values from 128-255. You're doing that in this line:
dbms_xslprocessor.clob2file( result, 'MYXML', ''||V_TABLE_NAME||'.xml',1);
You're passing 1 as the fourth parameter, csid. That value represents US7ASCII:
SQL> select nls_charset_name(1) from dual;
NLS_CHAR
--------
US7ASCII
Your XML is UTF-8, but specifying that with encoding="UTF-8" has no bearing on how the file is written. Any unrecognised characters are replaced with ?. So you might want to use the same setting for the file:
SQL> select nls_charset_id('UTF8') from dual;
NLS_CHARSET_ID('UTF8')
----------------------
871
So:
dbms_xslprocessor.clob2file( result, 'MYXML', ''||V_TABLE_NAME||'.xml',871);
or to be clearer:
dbms_xslprocessor.clob2file( result, 'MYXML', ''||V_TABLE_NAME||'.xml',
nls_charset_id('UTF8'));
But leaving it as the default might be OK - by not specifying csid at all, or by explicitly setting it to zero - depending on our database environment.
You mentioned you avoid ORA-31061 error if you "replace all ASCIICHAR (0-30) like ♂ : 11 ♀ : 12 ♫ : 14 ☼ : 15 ► : 16 ◄ : 17 ↕ : 18 ‼ : 19 ¶ : 20". Those symbols aren't what you'd expect from ASCII, so your character set or client or something seems to be interpreting them differently.
I get the error with all the ASCII control characters, 0 through to 31, except the printable ones: 9, 10 or 13. But that's what's expected, the other characters in that range are not valid in XML 1.0:
U+0009, U+000A, U+000D: these are the only C0 controls accepted in XML 1.0;
The same page shows that more, but still not all, control characters are allowed in XML 1.1, but as far as I'm aware Oracle only supports 1.0. If you do really have control characters in your data you'll need to strip them (retaining tabs, new lines and carriage returns); the rest would be meaningless in the final XML anyway, and perhaps of limited use in your existing data. I'm not sure if this is real data, or if you'd generated those values as a test.
I have one stored Procedure P_FP_GET_PATTERN.I expect output in following format
PATTERN_ID |PATTERN_NAME | SHIFT
1 Pattern 1 A,B,C,B,A
2 Pattern 2 C,B,A,C
I'm getting this output in SYS_REFCURSOR ALL_RESULT_SET.I need to pass it in XML format.But the moment i put that output in ALL_RESULT_SET_XML which is my out parameter of stored procedure of XMLTYPE using ALL_RESULT_SET_XML:= XMLTYPE(ALL_RESULT_SET);
Im getting an error as
Error encountered: ORA-31061: XDB error: special char to escaped char conversion failed.
I`m getting this error due to column shown using LISTAGG() function.Anybody can please tell me how to handle this ?
My stored Procedure
create or replace
PROCEDURE P_FP_GET_PATTERN
(
ALL_RESULT_SET_XML OUT XMLTYPE,
P_MESSAGE_ALL OUT VARCHAR2
)
AS
V_ERROR VARCHAR2(2000);
ALL_RESULT_SET SYS_REFCURSOR;
BEGIN
OPEN ALL_RESULT_SET FOR
SELECT PM.PATTERN_ID ,PM.PATTERN_NAME,
LISTAGG(SM.SHIFT_NUMBER) WITHIN GROUP (ORDER BY PD.INSTANCE_DAY) "SHIFT"
FROM T_FP_PATTERN_MASTER PM,
T_FP_PATTERN_DETAILS PD,
T_FP_SHIFT_MASTER SM
WHERE SM.SHIFT_ID= PD.SHIFT_ID
AND PM.PATTERN_ID = PD.PATTERN_ID
GROUP BY PM.PATTERN_NAME,PM.PATTERN_ID;
--Adding output in XML output parameter
ALL_RESULT_SET_XML:= XMLTYPE(ALL_RESULT_SET);
EXCEPTION
WHEN OTHERS
THEN
V_ERROR := SUBSTR(SQLERRM,1,1000);
P_MESSAGE_ALL := 'Error encountered: '||V_ERROR ;
END;