How to set large string as param without getting ORA-01460: unimplemented or unreasonable conversion error? - sql

In spring-boot using namedParameterJdbcTemplate (Oracle db version 12 and odbc8 driver 12.2)
I am getting the following error while executing a SELECT query bound with a parameter larger than 4000 character whereas update queries working fine.
ORA-01460: unimplemented or unreasonable conversion requested
The unit test I am trying to execute;
#Test
public void testSqlSelectQueryLargeStringParameter() {
MapSqlParameterSource params = new MapSqlParameterSource();
params.addValue("param", theLargeStr);
// #formatter:off
String sql =
"SELECT NULL id, NULL text FROM DUAL WHERE 'X' != :param ";
// #formatter:on
namedParameterJdbcTemplate.queryForRowSet(sql, params);
}
Is there any way to set this large param via MapSqlParameterSource?

I am #ahmet-orhan 's colleague, we've found a solution.
Thanks #kfinity for your suggestion, It is working for insert and update but we are still getting this error when we set clob or blob as "paremeter" in select statements.
If using a driver that supports JDBC4.0, the right solution is create a DefaultLobHandler and set streamAsLob or createTemporaryLob to true.
MapSqlParameterSource params = new MapSqlParameterSource();
String myString = "";
for (int i = 0; i < MAX_CLOB_BLOB_SIZE_IN_SELECT; i++) {
myString = myString + "1";
}
DefaultLobHandler lobHandler = new DefaultLobHandler();
lobHandler.setStreamAsLob(true);
params.addValue("param", new SqlLobValue(myString, lobHandler), Types.CLOB);
// #formatter:off
String sql =
"SELECT 1 id FROM DUAL WHERE :param IS NOT NULL ";
// #formatter:on
Integer id = namedParameterJdbcTemplate.queryForObject(sql, params, Integer.class);
We prefer streamAsLob but to be honest we have no idea which one is better.

This comment points out that ORA-01460 in JDBC queries is the same as "ORA-01704: string literal too long". (You can't have string literals longer than 4000 characters.) Maybe try this solution?
params.addValue("param", theLargeStr, Types.CLOB);
Although also != won't work for clob comparison, so you'll also need to change your query to
SELECT NULL id, NULL text FROM DUAL WHERE dbms_lob.compare('X',:param) != 0

Related

SQL String or binary data would be truncated

I'm trying to search the max ID in the database, then increment it by 1 for each new entry as shown below:
var query2 = dbb.Database.SqlQuery<patient_visit>("SELECT MAX(CAST(SUBSTRING(pvid, 3, 10) AS int)) FROM patient_visit");
if (query2 != null)
{
objDetails.pvid = query2.ToString();
objDetails.pvid += 1;
objDetails.pvid = "PV" + objDetails.pvid;
}
string sql = "INSERT INTO patient_visit (pvid,paid) " +
"VALUES('" + objDetails.pvid + "', '" + paid + "')";
But when i try to insert it in the database, it gives out error
An exception of type 'System.Data.SqlClient.SqlException' occurred in EntityFramework.SqlServer.dll but was not handled in user code
Additional information: String or binary data would be truncated.
I've tried running the query in SQL Server and checked the value of 'pvid', it is 10 so after i include 'PV' to the integer pvid, it should be 12. So it should be fine, but why did i get that error? Please can anyone help me?
[Key]
[MaxLength(20), MinLength(12)]
public string pvid { get; set; }
When i replace the pvid with hard code ID, it works just fine. Why is this happening?
P/S: I know its not advisable to simply concatenate the input data with the query, but i've also tried querying using parameterized query but it gives same error.
string sql = "INSERT INTO patient_visit (pvid,paid) " +
"VALUES(#pvid, #paid)";
List<SqlParameter> parameterList = new List<SqlParameter>();
parameterList.Add(new SqlParameter("#pvid", objDetails.pvid));
parameterList.Add(new SqlParameter("#paid", paid));
SqlParameter[] parameters = parameterList.ToArray();
dbb.Database.ExecuteSqlCommand(sql, parameters);
Check the Field length of the column you are trying to put the data into.
I have had this problem before where the column was varchar(10) and I was trying to concatenate two char strings of 6 characters each. Making them 12 characters long. 2 longer than the target field.
Your problem might be similar.

Java executed statement not returning string data in resultset

I have a simple SQL code that returns one record but when I execute it from Java, it does not return the string portions of the record, only numerical. The fields are VARCHAR2 but do not get extracted into my resultset. Following is the code. The database connectivity portion has been edited out for posting in the forum but it does connect. I have also attached the output. Any guidance would be appreciated as my searches on the web have returned empty. -Greg
package testsql;
import java.sql.*;
public class TestSQL {
String SQLtracknbr;
int SQLtracklength;
int numberOfColumns;
String coltypename;
int coldispsize;
String SQLschemaname;
public static void main(String[] args)
throws ClassNotFoundException, SQLException
{
Class.forName("oracle.jdbc.driver.OracleDriver");
DriverManager.registerDriver (new oracle.jdbc.driver.OracleDriver());
String url = "jdbc:oracle:thin:#oracam.corp.mot.com:1522:oracam";
String SQLcode = "select DISTINCT tracking_number from sfc_unit_process_track where tracking_number = 'CAH15F6WW9'";
System.out.println(SQLcode);
Connection conn =
DriverManager.getConnection(url,"report","report");
conn.setAutoCommit(false);
try (Statement stmt = conn.createStatement(); ResultSet rset = stmt.executeQuery(SQLcode)) {
ResultSetMetaData rsmd = rset.getMetaData();
while (rset.next()) {
int numberOfColumns = rsmd.getColumnCount();
boolean b = rsmd.isSearchable(1);
String coltypename = rsmd.getColumnTypeName(1);
int coldispsize = rsmd.getColumnDisplaySize(1);
String SQLschemaname = rsmd.getSchemaName(1);
String SQLtracknbr = rset.getString(1);
int SQLtracklength = SQLtracknbr.length();
if (SQLtracknbr == null)
System.out.println("NULL**********************.");
else
System.out.println("NOT NULL.");
System.out.println("numberOfColumns = " + numberOfColumns);
System.out.println("column type = " + coltypename);
System.out.println("column display size = " + coldispsize);
System.out.println("tracking_number = " + SQLtracknbr);
System.out.println("track number length = " + SQLtracklength);
System.out.println("schema name = " + SQLschemaname);
}
}
System.out.println ("*******End of code*******");
}
}
The result of what is executed in Java is below:
run:
select DISTINCT tracking_number from sfc_unit_process_track where tracking_number = 'CAH15F6WW9'
NOT NULL.
numberOfColumns = 1
column type = VARCHAR2
column display size = 30
tracking_number =
track number length = 0
schema name =
*******End of code*******
BUILD SUCCESSFUL (total time: 0 seconds)
This seems to be caused by an incompatibility between the driver you're using, ojdbc7.jar, and the version of the database you're connecting to, 9i.
According to the JDBC FAQ section "What are the various supported Oracle database version vs JDBC compliant versions vs JDK version supported?", the JDK 7/8 driver ojdbc7.jar that you're using is only support for Oracle 12c.
Oracle generally only support client/server versions two release apart (see My Oracle Support note 207303.1), and the Oracle 12c and 9i client and server have never been supported either way around. JDBC is slightly different of course, but it may be related, as drivers are installed with the Oracle software.
You will have to upgrade your database to a supported version, or - perhaps more practically in the short term - use an earlier driver. The Wayback Machine snapshot of the JDBC FAQ from 2013 says the 11.2.0 JDBC drivers - which includes ojdbc6.jar and ojcbd5.jar - can talk to RDBMS 9.2.0. So either of those ought to work...

ResultSet coming as empty after executing query

I have a query
SELECT instance_guid FROM service_instances WHERE service_template_guid='E578F99360A86E4EE043C28DE50A1D84' AND service_family_name='TEST'
Directly executing this returns me
4FEFDE7671A760A8DC8FC63CFBFC8316
F2F9DF641D8E2CACC03175A7A628D51D
Now I am trying same code from JDBC.
PreparedStatement ps = null;
ResultSet rs = null;
try {
conn = executionContext.getConnection();
if (conn != null) {
ps = (PreparedStatement)conn.prepareStatement(query);
if (params == null) params = new Object[0];
for (int i=0;i<params.length;i++) {
if (params[i] instanceof Integer) {
ps.setInt(i+1, ((Integer)params[i]).intValue());
} else if (params[i] instanceof java.util.Date) {
((PreparedStatement)ps).setDATE(i+1, new oracle.sql.DATE((new java.sql.Timestamp(((Date)params[i]).getTime()))));
//ps.setObject(i+1, new oracle.sql.DATE(new Time(((Date)params[i]).getTime())));
} else {
if (params[i] == null) params[i] = "";
ps.setString(i+1, params[i].toString());
}
}
rs = ps.executeQuery();
I see params[0] =E578F99360A86E4EE043C28DE50A1D84 and params[1]=TEST
But the resultSet is empty and not getting the result.I debugged but not much help?
Can you please let me know Am i trying right?
In java its defined as below
final static private String INSTANCE_GUID_BY_TEMPLATE_GUID =
"SELECT instance_guid FROM service_instances WHERE service_template_guid=? AND service_family_name=? "
SERVICE_FAMILY_NAME NOT NULL VARCHAR2(256)
SERVICE_TEMPLATE_GUID NOT NULL RAW(16 BYTE)
First and foremost this breaks every sql mapping pattern I have ever seen.
String sql = "SELECT instance_guid FROM service_instances WHERE service_template_guid=? AND service_family_name=?";
PreparedStatement ps = null;
ResultSet rs = null;
try {
conn = executionContext.getConnection();
ps = conn.prepareStatement(sql);
ps.setString(1,guid);
ps.setString(2,family);
rs = ps.executeQuery();
while(rs.next(){...}
...
}
You should not be dynamically figuring out the data types as they come in, unless you are trying to write some code to port from database X to database Y.
UPDATE
I see you are using RAW as a datatype, from this post:
As described in the Oracle JDBC Developer's guide and reference 11g,
when using a RAW column, you can treat it as a BINARY or VARBINARY
JDBC type, which means you can use the JDBC standard methods
getBytes() and setBytes() which returns or accepts a byte[]. The other
options is to use the Oracle driver specific extensions getRAW() and
setRAW() which return or accept a oracle.sql.RAW. Using these two will
require you to unwrap and/or cast to the specific Oracle
implementation class.
Further from a code readability standpoint, your solution makes it painful for a new developer to take over. Far too often I see people making sql be "dynamic" when in reality 99% of the time you don't need this level of dynamic query building. It sounds good in most people's heads but it just causes pain and suffering in the SDLC.

Does Dapper support the like operator?

Using Dapper-dot-net...
The following yields no results in the data object:
var data = conn.Query(#"
select top 25
Term as Label,
Type,
ID
from SearchTerms
WHERE Term like '%#T%'",
new { T = (string)term });
However, when I just use a regular String Format like:
string QueryString = String.Format("select top 25 Term as Label, Type, ID from SearchTerms WHERE Term like '%{0}%'", term);
var data = conn.Query(QueryString);
I get 25 rows back in the collection. Is Dapper not correctly parsing the end of the parameter #T?
Try:
term = "whateverterm";
var encodeForLike = term => term.Replace("[", "[[]").Replace("%", "[%]");
string term = "%" + encodeForLike(term) + "%";
var data = conn.Query(#"
select top 25
Term as Label,
Type,
ID
from SearchTerms
WHERE Term like #term",
new { term });
There is nothing special about like operators, you never want your params inside string literals, they will not work, instead they will be interpreted as a string.
note
The hard-coded example in your second snippet is strongly discouraged, besides being a huge problem with sql injection, it can cause dapper to leak.
caveat
Any like match that is leading with a wildcard is not SARGable, which means it is slow and will require an index scan.
Yes it does. This simple solution has worked for me everytime:
db.Query<Remitente>("SELECT *
FROM Remitentes
WHERE Nombre LIKE #n", new { n = "%" + nombre + "%" })
.ToList();
Best way to use this to add concat function in query as it save in sql injecting as well, but concat function is only support above than sql 2012
string query = "SELECT * from country WHERE Name LIKE CONCAT('%',#name,'%');"
var results = connection.query<country>(query, new {name});
The answer from Sam wasn't working for me so after some testing I came up with using the SQLite CONCAT equivalent which seems to work:
string sql = "SELECT * FROM myTable WHERE Name LIKE '%' || #NAME || '%'";
var data = IEnumerable data = conn.Query(sql, new { NAME = Name });
Just to digress on Sam's answer, here is how I created two helper methods to make searches a bit easier using the LIKE operator.
First, creating a method for generating a parameterized query, this method uses dynamic: , but creating a strongly typed generic method should be more desired in many cases where you want static typing instead of dynamic.
public static dynamic ParameterizedQuery(this IDbConnection connection, string sql, Dictionary<string, object> parametersDictionary)
{
if (string.IsNullOrEmpty(sql))
{
return null;
}
string missingParameters = string.Empty;
foreach (var item in parametersDictionary)
{
if (!sql.Contains(item.Key))
{
missingParameters += $"Missing parameter: {item.Key}";
}
}
if (!string.IsNullOrEmpty(missingParameters))
{
throw new ArgumentException($"Parameterized query failed. {missingParameters}");
}
var parameters = new DynamicParameters(parametersDictionary);
return connection.Query(sql, parameters);
}
Then adding a method to create a Like search term that will work with Dapper.
public static string Like(string searchTerm)
{
if (string.IsNullOrEmpty(searchTerm))
{
return null;
}
Func<string, string> encodeForLike = searchTerm => searchTerm.Replace("[", "[[]").Replace("%", "[%]");
return $"%{encodeForLike(searchTerm)}%";
}
Example usage:
var sql = $"select * from products where ProductName like #ProdName";
var herringsInNorthwindDb = connection.ParameterizedQuery(sql, new Dictionary<string, object> { { "#ProdName", Like("sild") } });
foreach (var herring in herringsInNorthwindDb)
{
Console.WriteLine($"{herring.ProductName}");
}
And we get our sample data from Northwind DB:
I like this approach, since we get helper extension methods to do repetitive work.
My solution simple to this problem :
parameter.Add("#nomeCliente", dfNomeCliPesquisa.Text.ToUpper());
query = "SELECT * FROM cadastrocliente WHERE upper(nome) LIKE " + "'%" + dfNomeCliPesquisa.Text.ToUpper() + "%'";

error, string or binary data would be truncated when trying to insert

I am running data.bat file with the following lines:
Rem Tis batch file will populate tables
cd\program files\Microsoft SQL Server\MSSQL
osql -U sa -P Password -d MyBusiness -i c:\data.sql
The contents of the data.sql file is:
insert Customers
(CustomerID, CompanyName, Phone)
Values('101','Southwinds','19126602729')
There are 8 more similar lines for adding records.
When I run this with start > run > cmd > c:\data.bat, I get this error message:
1>2>3>4>5>....<1 row affected>
Msg 8152, Level 16, State 4, Server SP1001, Line 1
string or binary data would be truncated.
<1 row affected>
<1 row affected>
<1 row affected>
<1 row affected>
<1 row affected>
<1 row affected>
Also, I am a newbie obviously, but what do Level #, and state # mean, and how do I look up error messages such as the one above: 8152?
From #gmmastros's answer
Whenever you see the message....
string or binary data would be truncated
Think to yourself... The field is NOT big enough to hold my data.
Check the table structure for the customers table. I think you'll find that the length of one or more fields is NOT big enough to hold the data you are trying to insert. For example, if the Phone field is a varchar(8) field, and you try to put 11 characters in to it, you will get this error.
I had this issue although data length was shorter than the field length.
It turned out that the problem was having another log table (for audit trail), filled by a trigger on the main table, where the column size also had to be changed.
In one of the INSERT statements you are attempting to insert a too long string into a string (varchar or nvarchar) column.
If it's not obvious which INSERT is the offender by a mere look at the script, you could count the <1 row affected> lines that occur before the error message. The obtained number plus one gives you the statement number. In your case it seems to be the second INSERT that produces the error.
Just want to contribute with additional information: I had the same issue and it was because of the field wasn't big enough for the incoming data and this thread helped me to solve it (the top answer clarifies it all).
BUT it is very important to know what are the possible reasons that may cause it.
In my case i was creating the table with a field like this:
Select '' as Period, * From Transactions Into #NewTable
Therefore the field "Period" had a length of Zero and causing the Insert operations to fail. I changed it to "XXXXXX" that is the length of the incoming data and it now worked properly (because field now had a lentgh of 6).
I hope this help anyone with same issue :)
Some of your data cannot fit into your database column (small). It is not easy to find what is wrong. If you use C# and Linq2Sql, you can list the field which would be truncated:
First create helper class:
public class SqlTruncationExceptionWithDetails : ArgumentOutOfRangeException
{
public SqlTruncationExceptionWithDetails(System.Data.SqlClient.SqlException inner, DataContext context)
: base(inner.Message + " " + GetSqlTruncationExceptionWithDetailsString(context))
{
}
/// <summary>
/// PArt of code from following link
/// http://stackoverflow.com/questions/3666954/string-or-binary-data-would-be-truncated-linq-exception-cant-find-which-fiel
/// </summary>
/// <param name="context"></param>
/// <returns></returns>
static string GetSqlTruncationExceptionWithDetailsString(DataContext context)
{
StringBuilder sb = new StringBuilder();
foreach (object update in context.GetChangeSet().Updates)
{
FindLongStrings(update, sb);
}
foreach (object insert in context.GetChangeSet().Inserts)
{
FindLongStrings(insert, sb);
}
return sb.ToString();
}
public static void FindLongStrings(object testObject, StringBuilder sb)
{
foreach (var propInfo in testObject.GetType().GetProperties())
{
foreach (System.Data.Linq.Mapping.ColumnAttribute attribute in propInfo.GetCustomAttributes(typeof(System.Data.Linq.Mapping.ColumnAttribute), true))
{
if (attribute.DbType.ToLower().Contains("varchar"))
{
string dbType = attribute.DbType.ToLower();
int numberStartIndex = dbType.IndexOf("varchar(") + 8;
int numberEndIndex = dbType.IndexOf(")", numberStartIndex);
string lengthString = dbType.Substring(numberStartIndex, (numberEndIndex - numberStartIndex));
int maxLength = 0;
int.TryParse(lengthString, out maxLength);
string currentValue = (string)propInfo.GetValue(testObject, null);
if (!string.IsNullOrEmpty(currentValue) && maxLength != 0 && currentValue.Length > maxLength)
{
//string is too long
sb.AppendLine(testObject.GetType().Name + "." + propInfo.Name + " " + currentValue + " Max: " + maxLength);
}
}
}
}
}
}
Then prepare the wrapper for SubmitChanges:
public static class DataContextExtensions
{
public static void SubmitChangesWithDetailException(this DataContext dataContext)
{
//http://stackoverflow.com/questions/3666954/string-or-binary-data-would-be-truncated-linq-exception-cant-find-which-fiel
try
{
//this can failed on data truncation
dataContext.SubmitChanges();
}
catch (SqlException sqlException) //when (sqlException.Message == "String or binary data would be truncated.")
{
if (sqlException.Message == "String or binary data would be truncated.") //only for EN windows - if you are running different window language, invoke the sqlException.getMessage on thread with EN culture
throw new SqlTruncationExceptionWithDetails(sqlException, dataContext);
else
throw;
}
}
}
Prepare global exception handler and log truncation details:
protected void Application_Error(object sender, EventArgs e)
{
Exception ex = Server.GetLastError();
string message = ex.Message;
//TODO - log to file
}
Finally use the code:
Datamodel.SubmitChangesWithDetailException();
Another situation in which you can get this error is the following:
I had the same error and the reason was that in an INSERT statement that received data from an UNION, the order of the columns was different from the original table. If you change the order in #table3 to a, b, c, you will fix the error.
select a, b, c into #table1
from #table0
insert into #table1
select a, b, c from #table2
union
select a, c, b from #table3
on sql server you can use SET ANSI_WARNINGS OFF like this:
using (SqlConnection conn = new SqlConnection("Data Source=XRAYGOAT\\SQLEXPRESS;Initial Catalog='Healthy Care';Integrated Security=True"))
{
conn.Open();
using (var trans = conn.BeginTransaction())
{
try
{
using cmd = new SqlCommand("", conn, trans))
{
cmd.CommandText = "SET ANSI_WARNINGS OFF";
cmd.ExecuteNonQuery();
cmd.CommandText = "YOUR INSERT HERE";
cmd.ExecuteNonQuery();
cmd.Parameters.Clear();
cmd.CommandText = "SET ANSI_WARNINGS ON";
cmd.ExecuteNonQuery();
trans.Commit();
}
}
catch (Exception)
{
trans.Rollback();
}
}
conn.Close();
}
I had the same issue. The length of my column was too short.
What you can do is either increase the length or shorten the text you want to put in the database.
Also had this problem occurring on the web application surface.
Eventually found out that the same error message comes from the SQL update statement in the specific table.
Finally then figured out that the column definition in the relating history table(s) did not map the original table column length of nvarchar types in some specific cases.
I had the same problem, even after increasing the size of the problematic columns in the table.
tl;dr: The length of the matching columns in corresponding Table Types may also need to be increased.
In my case, the error was coming from the Data Export service in Microsoft Dynamics CRM, which allows CRM data to be synced to an SQL Server DB or Azure SQL DB.
After a lengthy investigation, I concluded that the Data Export service must be using Table-Valued Parameters:
You can use table-valued parameters to send multiple rows of data to a Transact-SQL statement or a routine, such as a stored procedure or function, without creating a temporary table or many parameters.
As you can see in the documentation above, Table Types are used to create the data ingestion procedure:
CREATE TYPE LocationTableType AS TABLE (...);
CREATE PROCEDURE dbo.usp_InsertProductionLocation
#TVP LocationTableType READONLY
Unfortunately, there is no way to alter a Table Type, so it has to be dropped & recreated entirely. Since my table has over 300 fields (😱), I created a query to facilitate the creation of the corresponding Table Type based on the table's columns definition (just replace [table_name] with your table's name):
SELECT 'CREATE TYPE [table_name]Type AS TABLE (' + STRING_AGG(CAST(field AS VARCHAR(max)), ',' + CHAR(10)) + ');' AS create_type
FROM (
SELECT TOP 5000 COLUMN_NAME + ' ' + DATA_TYPE
+ IIF(CHARACTER_MAXIMUM_LENGTH IS NULL, '', CONCAT('(', IIF(CHARACTER_MAXIMUM_LENGTH = -1, 'max', CONCAT(CHARACTER_MAXIMUM_LENGTH,'')), ')'))
+ IIF(DATA_TYPE = 'decimal', CONCAT('(', NUMERIC_PRECISION, ',', NUMERIC_SCALE, ')'), '')
AS field
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = '[table_name]'
ORDER BY ORDINAL_POSITION) AS T;
After updating the Table Type, the Data Export service started functioning properly once again! :)
When I tried to execute my stored procedure I had the same problem because the size of the column that I need to add some data is shorter than the data I want to add.
You can increase the size of the column data type or reduce the length of your data.
A 2016/2017 update will show you the bad value and column.
A new trace flag will swap the old error for a new 2628 error and will print out the column and offending value. Traceflag 460 is available in the latest cumulative update for 2016 and 2017:
https://support.microsoft.com/en-sg/help/4468101/optional-replacement-for-string-or-binary-data-would-be-truncated
Just make sure that after you've installed the CU that you enable the trace flag, either globally/permanently on the server:
...or with DBCC TRACEON:
https://learn.microsoft.com/en-us/sql/t-sql/database-console-commands/dbcc-traceon-trace-flags-transact-sql?view=sql-server-ver15
Another situation, in which this error may occur is in
SQL Server Management Studio. If you have "text" or "ntext" fields in your table,
no matter what kind of field you are updating (for example bit or integer).
Seems that the Studio does not load entire "ntext" fields and also updates ALL fields instead of the modified one.
To solve the problem, exclude "text" or "ntext" fields from the query in Management Studio
This Error Comes only When any of your field length is greater than the field length specified in sql server database table structure.
To overcome this issue you have to reduce the length of the field Value .
Or to increase the length of database table field .
If someone is encountering this error in a C# application, I have created a simple way of finding offending fields by:
Getting the column width of all the columns of a table where we're trying to make this insert/ update. (I'm getting this info directly from the database.)
Comparing the column widths to the width of the values we're trying to insert/ update.
Assumptions/ Limitations:
The column names of the table in the database match with the C# entity fields. For eg: If you have a column like this in database:
You need to have your Entity with the same column name:
public class SomeTable
{
// Other fields
public string SourceData { get; set; }
}
You're inserting/ updating 1 entity at a time. It'll be clearer in the demo code below. (If you're doing bulk inserts/ updates, you might want to either modify it or use some other solution.)
Step 1:
Get the column width of all the columns directly from the database:
// For this, I took help from Microsoft docs website:
// https://learn.microsoft.com/en-us/dotnet/api/system.data.sqlclient.sqlconnection.getschema?view=netframework-4.7.2#System_Data_SqlClient_SqlConnection_GetSchema_System_String_System_String___
private static Dictionary<string, int> GetColumnSizesOfTableFromDatabase(string tableName, string connectionString)
{
var columnSizes = new Dictionary<string, int>();
using (var connection = new SqlConnection(connectionString))
{
// Connect to the database then retrieve the schema information.
connection.Open();
// You can specify the Catalog, Schema, Table Name, Column Name to get the specified column(s).
// You can use four restrictions for Column, so you should create a 4 members array.
String[] columnRestrictions = new String[4];
// For the array, 0-member represents Catalog; 1-member represents Schema;
// 2-member represents Table Name; 3-member represents Column Name.
// Now we specify the Table_Name and Column_Name of the columns what we want to get schema information.
columnRestrictions[2] = tableName;
DataTable allColumnsSchemaTable = connection.GetSchema("Columns", columnRestrictions);
foreach (DataRow row in allColumnsSchemaTable.Rows)
{
var columnName = row.Field<string>("COLUMN_NAME");
//var dataType = row.Field<string>("DATA_TYPE");
var characterMaxLength = row.Field<int?>("CHARACTER_MAXIMUM_LENGTH");
// I'm only capturing columns whose Datatype is "varchar" or "char", i.e. their CHARACTER_MAXIMUM_LENGTH won't be null.
if(characterMaxLength != null)
{
columnSizes.Add(columnName, characterMaxLength.Value);
}
}
connection.Close();
}
return columnSizes;
}
Step 2:
Compare the column widths with the width of the values we're trying to insert/ update:
public static Dictionary<string, string> FindLongBinaryOrStringFields<T>(T entity, string connectionString)
{
var tableName = typeof(T).Name;
Dictionary<string, string> longFields = new Dictionary<string, string>();
var objectProperties = GetProperties(entity);
//var fieldNames = objectProperties.Select(p => p.Name).ToList();
var actualDatabaseColumnSizes = GetColumnSizesOfTableFromDatabase(tableName, connectionString);
foreach (var dbColumn in actualDatabaseColumnSizes)
{
var maxLengthOfThisColumn = dbColumn.Value;
var currentValueOfThisField = objectProperties.Where(f => f.Name == dbColumn.Key).First()?.GetValue(entity, null)?.ToString();
if (!string.IsNullOrEmpty(currentValueOfThisField) && currentValueOfThisField.Length > maxLengthOfThisColumn)
{
longFields.Add(dbColumn.Key, $"'{dbColumn.Key}' column cannot take the value of '{currentValueOfThisField}' because the max length it can take is {maxLengthOfThisColumn}.");
}
}
return longFields;
}
public static List<PropertyInfo> GetProperties<T>(T entity)
{
//The DeclaredOnly flag makes sure you only get properties of the object, not from the classes it derives from.
var properties = entity.GetType()
.GetProperties(System.Reflection.BindingFlags.Public
| System.Reflection.BindingFlags.Instance
| System.Reflection.BindingFlags.DeclaredOnly)
.ToList();
return properties;
}
Demo:
Let's say we're trying to insert someTableEntity of SomeTable class that is modeled in our app like so:
public class SomeTable
{
[Key]
public long TicketID { get; set; }
public string SourceData { get; set; }
}
And it's inside our SomeDbContext like so:
public class SomeDbContext : DbContext
{
public DbSet<SomeTable> SomeTables { get; set; }
}
This table in Db has SourceData field as varchar(16) like so:
Now we'll try to insert value that is longer than 16 characters into this field and capture this information:
public void SaveSomeTableEntity()
{
var connectionString = "server=SERVER_NAME;database=DB_NAME;User ID=SOME_ID;Password=SOME_PASSWORD;Connection Timeout=200";
using (var context = new SomeDbContext(connectionString))
{
var someTableEntity = new SomeTable()
{
SourceData = "Blah-Blah-Blah-Blah-Blah-Blah"
};
context.SomeTables.Add(someTableEntity);
try
{
context.SaveChanges();
}
catch (Exception ex)
{
if (ex.GetBaseException().Message == "String or binary data would be truncated.\r\nThe statement has been terminated.")
{
var badFieldsReport = "";
List<string> badFields = new List<string>();
// YOU GOT YOUR FIELDS RIGHT HERE:
var longFields = FindLongBinaryOrStringFields(someTableEntity, connectionString);
foreach (var longField in longFields)
{
badFields.Add(longField.Key);
badFieldsReport += longField.Value + "\n";
}
}
else
throw;
}
}
}
The badFieldsReport will have this value:
'SourceData' column cannot take the value of
'Blah-Blah-Blah-Blah-Blah-Blah' because the max length it can take is
16.
Kevin Pope's comment under the accepted answer was what I needed.
The problem, in my case, was that I had triggers defined on my table that would insert update/insert transactions into an audit table, but the audit table had a data type mismatch where a column with VARCHAR(MAX) in the original table was stored as VARCHAR(1) in the audit table, so my triggers were failing when I would insert anything greater than VARCHAR(1) in the original table column and I would get this error message.
I used a different tactic, fields that are allocated 8K in some places. Here only about 50/100 are used.
declare #NVPN_list as table
nvpn varchar(50)
,nvpn_revision varchar(5)
,nvpn_iteration INT
,mpn_lifecycle varchar(30)
,mfr varchar(100)
,mpn varchar(50)
,mpn_revision varchar(5)
,mpn_iteration INT
-- ...
) INSERT INTO #NVPN_LIST
SELECT left(nvpn ,50) as nvpn
,left(nvpn_revision ,10) as nvpn_revision
,nvpn_iteration
,left(mpn_lifecycle ,30)
,left(mfr ,100)
,left(mpn ,50)
,left(mpn_revision ,5)
,mpn_iteration
,left(mfr_order_num ,50)
FROM [DASHBOARD].[dbo].[mpnAttributes] (NOLOCK) mpna
I wanted speed, since I have 1M total records, and load 28K of them.
This error may be due to less field size than your entered data.
For e.g. if you have data type nvarchar(7) and if your value is 'aaaaddddf' then error is shown as:
string or binary data would be truncated
You simply can't beat SQL Server on this.
You can insert into a new table like this:
select foo, bar
into tmp_new_table_to_dispose_later
from my_table
and compare the table definition with the real table you want to insert the data into.
Sometime it's helpful sometimes it's not.
If you try inserting in the final/real table from that temporary table it may just work (due to data conversion working differently than SSMS for example).
Another alternative is to insert the data in chunks, instead of inserting everything immediately you insert with top 1000 and you repeat the process, till you find a chunk with an error. At least you have better visibility on what's not fitting into the table.