ORA-01704: string literal too long when migrating database - sql

I migrate some data from a database and get it in a data.xml file
Example:
<insert tableName="ORG_CUSTOMER">
<column name="My-Column" value="a very long string.."/>
</insert>
I then execute it with a cmd command
call ..\liquibase\liquibase-bin\liquibase.bat --changeLogFile=%DATA% --defaultSchemaName=NAME_DBA --defaultsFile=liquibase-local.properties update
And because the value sometimes is longer than 4000 characters i get the error:
Unexpected error running Liquibase: ORA-01704: string literal too long
The column is of value clob. The autogenerated xml file contains a lot of inserts I would prefer to not edit it manually. I read something that making it a prepared statement or using pl/sql will work but I don't now how to do that?

Oracle - by default - has a limit of 4000 byte for string literals.
If you are on 12.1 or newer you could raise that limit to 32k.
If you are using an old version or can't change the parameter, you can't use Liquibase's <insert> change.
You could try the <loadData> change and hope it uses a PreparedStatement, in that case the limit doesn't apply:
http://www.liquibase.org/documentation/changes/load_data.html

Without the code of liquibase.bat it is difficult to say but.
From the documentation on literals:
Text literals have properties of both the CHAR and VARCHAR2 datatypes:
Within expressions and conditions, Oracle treats text literals as though they have the datatype CHAR by comparing them using blank-padded comparison semantics.
A text literal can have a maximum length of 4000 bytes.
I would guess that the script is naively converting the XML from:
<insert tableName="ORG_CUSTOMER">
<column name="My-Column" value="a very long string.."/>
</insert>
To
INSERT INTO {tablename} ( "{column.name}" ) VALUES ( '{column.value}' );
and this will fail when the text literal has more than 4000 bytes.
To fix this you will need to edit the script to handle CLOB values with more than 4000 bytes (where you cannot use a text literal).

Related

Difference between N'String' vs U'String' literals in Oracle

What is the meaning and difference between these queries?
SELECT U'String' FROM dual;
and
SELECT N'String' FROM dual;
In this answer i will try to provide informations from official resources
(1) The N'' text Literal
N'' is used to convert a string to NCHAR or NVARCHAR2 datatype
According to this Oracle documentation Oracle - Literals
The syntax of text literals is as follows:
where N or n specifies the literal using the national character set (NCHAR or NVARCHAR2 data).
Also in this second article Oracle - Datatypes
The N'String' is used to convert a string to NCHAR datatype
From the article listed above:
The following example compares the translated_description column of the pm.product_descriptions table with a national character set string:
SELECT translated_description FROM product_descriptions
WHERE translated_name = N'LCD Monitor 11/PM';
(2) The U'' Literal
U'' is used to handle the SQL NCHAR String Literals in Oracle Call Interface (OCI)
Based on this Oracle documentation Programming with Unicode
The Oracle Call Interface (OCI) is the lowest level API that the rest of the client-side database access products use. It provides a flexible way for C/C++ programs to access Unicode data stored in SQL CHAR and NCHAR datatypes. Using OCI, you can programmatically specify the character set (UTF-8, UTF-16, and others) for the data to be inserted or retrieved. It accesses the database through Oracle Net.
OCI is the lowest-level API for accessing a database, so it offers the best possible performance.
Handling SQL NCHAR String Literals in OCI
You can switch it on by setting the environment variable ORA_NCHAR_LITERAL_REPLACE to TRUE. You can also achieve this behavior programmatically by using the OCI_NCHAR_LITERAL_REPLACE_ON and OCI_NCHAR_LITERAL_REPLACE_OFF modes in OCIEnvCreate() and OCIEnvNlsCreate(). So, for example, OCIEnvCreate(OCI_NCHAR_LITERAL_REPLACE_ON) turns on NCHAR literal replacement, while OCIEnvCreate(OCI_NCHAR_LITERAL_REPLACE_OFF) turns it off.
[...] Note that, when the NCHAR literal replacement is turned on, OCIStmtPrepare and OCIStmtPrepare2 will transform N' literals with U' literals in the SQL text and store the resulting SQL text in the statement handle. Thus, if the application uses OCI_ATTR_STATEMENT to retrieve the SQL text from the OCI statement handle, the SQL text will return U' instead of N' as specified in the original text.
(3) Answer for your question
From datatypes perspective, there is not difference between both queries provided
N'string' just returns the string as NCHAR type.
U'string' returns also NCHAR type, however it does additional processing to the string: it replaces \\ with \ and \xxxx with Unicode code point U+xxxx, where xxxx are 4 hexadecimal digits. This is similar to UNISTR('string'), the difference is that the latter returns NVARCHAR2.
U' literals are useful when you want to have a Unicode string independent from encoding and NLS settings.
Example:
select n'\€', u'\\\20ac', n'\\\20ac' from dual;
N'\€' U'\\\20AC' N'\\\20AC'
----- ---------- ----------
\€ \€ \\\20ac
when using N' we denote that given datatype is NCHAR or NVARCHAR.
U' is used to denote unicode
The documented N'' literals are the same as standard character literals ('') except that their data type is NVARCHAR2 and not VARCHAR2. It is important to note that the characters in these literals, together with the entire SQL statement, are converted from the client character set to the database character set when transmitted to the server. All characters from the literals that are not supported by the database character set are lost.
The data type of the undocumented U'' literals is also NVARCHAR2. The content of a U'' literal is interpreted like the input to the SQL UNISTR function. That is, each character sequence \xxxx, where each x is one hex digit, is interpreted as a UTF-16 code point U+xxxx. I am not sure why the U'' literals are undocumented. I can only guess. They are used internally by the NCHAR literal replacement feature, which, when enable on a client, automatically translates N'' literals to U'' literals. This prevents the mentioned data loss due to character set conversion and enables literal Unicode data to be provided for NVARCHAR2 columns even if the database character set is not Unicode.
The two queries in this thread's question are generally not equivalent because the literal text would be interpreted differently. However, if no backslash is present in the literals, no difference can be observed.

[Excel Destination [28]] Error: An error occurred while setting up a binding for the column. The binding status was "DT_NTEXT"

I'm working on ssis package which exports data from SQL Server to Excel. I had a problem converting non-unicode to unicode string data types. So I created a derived Column task and converted to Unicode string [DT_WSTR] 4 columns which have a type Varchar(40) in SQL Server table. It worked with these columns. But I also have a Description column of type varchar(max) and I tried to convert it to Unicode text stream [DT_NTEXT]. It did not work.
If your source is SQL Server (as you said), you can convert it directly in your SQL Query
SELECT
CONVERT(NVARCHAR(40), 'att1')
,CONVERT(NTEXT, 'att2')
Convert your VARCHAR into NVARCHAR
Convert your TEXT into NTEXT
it's faster.
P.S. To test it (Do not forget to delete or reset your previous OLE DB Input component) -> It will be forced to reevaluate your datatype
Does it help you?
The only thing that worked was to cast a Description column in Stored Procedure as varchar(1000). I checked the max length of this field and it was about 300 characters. So I made it varchar(1000) and in Derived column Unicode string [DT_WSTR]. This was a workaround, but I still want to know how to make it in ssis package without converting data type in Stored Procedure.

liquibase error String too long more than 4000 charchater

I have question regarding liquibase.I am getting an error while inserting clob type data.String literal is too long.I have mentioned as follows
column name="help_item_text" type="clob" value="String too long more than 4000 charchaters"
But no luck still same.
Normally Liquibase uses standard SQL statements so that there is no difference between updateSql and update modes. For CLOB fields, that can run into problems when the total SQL length gets to be longer than the database's SQL parser can handle.
There is a valueClobFile attribute on column that allows you to save the long value to a file and then reference it from the changelog file. This gets the large value out of your changelog file and also tells Liquibase it needs to use a prepared statement.
<column name="help_item_text" type="clob" valueClobFile="String too long more than 4000 characters">

Losing special characters on insert

I am using oracle 11g and trying to insert a string containing special UTF8 characters eg '(ε- c'. The NLS character sets for the databse are...
NLS_NCHAR_CHARACTERSET AL16UTF16
NLS_CHARACTERSET WE8ISO8859P1
when I copy and paste the above string into a NVARCHAR field it works fine.
if I execute the below I get an upside down question mark in the field
insert into title_debug values ('(ε- c');
where title debug table consists of a single NVARCHAR2(100) field called title.
I have attempted to assign this string to a NVARCHAR2(100) variable then iserting this. And also attempted all the different CAST / CONVERT ect functions I can find and nothing is working.
Any assistance would be greatly appreciated.
UPDATE
I have executed
select dump(title, 1016), dump(title1, 1016)
into v_title, v_title1
from dual
where title is the string passed in as a varchar and title1 is the string passed in as a NVarchar.
Unsuprisingly the encodings come through as WE8ISO8859P1 and AL16UTF16. but on both the ε comes through as hex 'BF'. This is the upside down Question mark.
My only thought left is to try and pass this through as a raw and then do something with it. However I have not yet been able to figure out how to convert the string into a acceptable format with XQuery (OSB).
Continued thanks for assistance.
Our DBA found the solution to this issue. The answer lay in a setting on the dbc connection on the bus to tell it to convert utf8 to NChar.
On The connection pool page add the following lines to the Properties box.
oracle.jdbc.convertNcharLiterals=true
oracle.jdbc.defaultNchar=true
this will allow you to be able to insert into NVarchar2 fields while maintaining the utf8 characters.
Cheers
I did a test with '(ε- c'. And i have not encountered any problems.
If you can i advice you to change your character set for :
NLS_CHARACTERSET AL32UTF8
NLS_NCHAR_CHARACTERSET AL16UTF16
And Oracle recommendation for all new deployment is the Unicode character set AL32UTF8.
Because it's flexible globalization support and is a universal character set
reg,
First verify the data is being stored correctly, then use the correct NLS_LANG settings. See my answer to this question:
when insert persian character in oracle db i see the question mark

Unable to insert beyond 4000 character

I am trying to insert a value(eg O20040601120101SYSONEBNPENDING20040601101010Y00405) into oracle db with the data type long, But I am getting an error saying
Error report:
SQL Error: ORA-01704: string literal too long
01704. 00000 - "string literal too long"
*Cause: The string literal is longer than 4000 characters.
*Action: Use a string literal of at most 4000 characters.
Longer values may only be entered using bind variables.
I tried using clob also but not able to solve this please help me in this.
The maximum for one time insertion is 4000 characters (the maximum string literal in Oracle). However you can use the lob function dbms_lob.append() to append chunks of (maximum)
4000 characters to the clob.
See Here
This is useful
As stated in a previous comment, you should not use the LONG datatype. It's been deprecated many years ago...
If you are using Oracle 12c, remember that you can use VARCHAR2 datatype too. It's been extended to store up to 32767 bytes!