Nhibernate Criteria: 'select max(id)...' - nhibernate

Can I use a Criteria to execute a t-sql command to select the max value for a column in a table?
'select #cus_id = max(id) + 1 from customers'
Ta
Ollie

Use Projection:
session.CreateCriteria(typeof(Customer))
.SetProjection( Projections.Max("Id") )
. UniqueResult();

Max(id) + 1 is a very bad way to generate ids. If that's your goal, find another way to generate ids.
Edit: in answer to LnDCobra:
it's bad because it's hard to make sure that the max(id) you got is still the max(id) when you do the insert. If another process inserts a row, your insert will have the same id, and your insert will fail. (Or, conversely, the other process's insert will fail if your insert happened first.)
To prevent this, you have to prevent any other inserts/make your get and subsequent insert atomic, which generally means locking the table, which will hurt performance.
If you only lock against writes, the other process gets max(id), which is the same max(id) you got. You do your insert and release the lock, it inserts a duplicate id and fails. Or it tries to lock too, in which case it waits on you. If you lock against reads too, everybody waits on you. If it locks against writes also, then it doesn't insert the duplicate id, but it does wait on your read and your write.
(And it breaks encapsulation: you should let the rdbms figure out its ids, not the client programs that connect to it.)
Generally, this strategy will either:
* break
* require a bunch of "plumbing" code to make it work
* significantly reduce performance
* or all three
and it will be slower, less robust, and require more hard to maintain code than just using the RDBMS's built in sequences or generated autoincrement ids.

Best approach is to make additional Sequences table.
Where you can maintain sequence target and value.
public class Sequence : Entity
{
public virtual long? OwnerId { get; set; }
public virtual SequenceTarget SequenceTarget { get; set; }
public virtual bool IsLocked { get; set; }
public virtual long Value { get; set; }
public void GenerateNextValue()
{
Value++;
}
}
public class SequenceTarget : Entity
{
public virtual string Name { get; set; }
}
public long GetNewSequenceValueForZZZZ(long ZZZZId)
{
var target =
Session
.QueryOver<SequenceTarget>()
.Where(st => st.Name == "DocNumber")
.SingleOrDefault();
if (target == null)
{
throw new EntityNotFoundException(typeof(SequenceTarget));
}
return GetNewSequenceValue(ZZZZId, target);
}
protected long GetNewSequenceValue(long? ownerId, SequenceTarget target)
{
var seqQry =
Session
.QueryOver<Sequence>()
.Where(seq => seq.SequenceTarget == target);
if (ownerId.HasValue)
{
seqQry.Where(seq => seq.OwnerId == ownerId.Value);
}
var sequence = seqQry.SingleOrDefault();
if (sequence == null)
{
throw new EntityNotFoundException(typeof(Sequence));
}
// re-read sequence, if it was in session
Session.Refresh(sequence);
// update IsLocked field, so we acuire lock on record
// configure dynamic update , so only 1 field is being updated
sequence.IsLocked = !sequence.IsLocked;
Session.Update(sequence);
// force update to db
Session.Flush();
// now we gained block - re-read record.
Session.Refresh(sequence);
// generate new value
sequence.GenerateNextValue();
// set back dummy filed
sequence.IsLocked = !sequence.IsLocked;
// update sequence & force changes to DB
Session.Update(sequence);
Session.Flush();
return sequence.Value;
}
OwnerId - when you need to maintain different sequences for same entity, based on some kind of owner. For example you need to maintain numbering for document within contract, then OwnerId will be = contractId

Related

How can I validate if a record can be added at the SQL level using Entity Framework Core

If I want to make some checks before inserting a row into the database, I know that I can run the following code:
public bool BookSeat(int userId, string seatNumber)
{
If (IsSeatStillAvailable(seatNumber))
{
var ticket = new Ticket(userId, seatNumber);
_dbContext.Tickets(ticket);
_dbContext.SaveChanges();
return true;
}
return false;
}
private bool IsSeatStillAvailable(seatNumber)
{
var seatTaken = _dbcontext.Tickets.Any(w => w.seatNumber == seatNumber);
return !seatTaken;
}
This will do a call to the database to see if the seat is taken and then do a second call to book the seat. BUT in that time it might have already been booked.
I know in simple examples, I can create an index, but my use case is much more complex.
Is there a way that I can generate the SQL statement on the insert so that it can produce an ATOMIC transaction?
To produce something like (excuse the clumsy SQL):
IF (SELECT TOP 1 Count(*) FROM Tickets = 0)
BEGIN
INSERT INTO Tickets (UserId, SeatNumber)
VALUES (#UserId, #SeatNumber);
RETURN true
END;
RETURN false
What you are looking for is concurrency handling and optimistic locking :
https://learn.microsoft.com/en-us/ef/core/saving/concurrency?tabs=data-annotations

LINQ or Navigation Properties command to retrieve 1 to many data

I am looking for help with a LINQ SQL query please.
I have a blazor application that gets data from an Azure SQL database. I am seeking to get a dataset from the database for linking to a datagrid, where each row is a record from the main table joined with a record from the second table. The second table has millions of records, it needs to join one record which has the same key (securityId) and with the date being the record with the nominated date, or with the last date before the nominated date.
Because of the size of the 2nd file, I need an efficient query. Currently I am using the following, but I believe there must be more efficient ways to do it without the lag. Also tried Navigation Properties but couldn't get to work
reviewdateS is the date that I want the 2nd record to match or be the latest date prior to that date
result = (from cmpn in _ctx.MstarCompanies
join prcs in _ctx.MstarPrices
on cmpn.securityId equals prcs.securityId into cs
from c in cs.DefaultIfEmpty()
where c.date01 == reviewDateS
select new ClsMarketPrices { })
Following are the 3 relevant classes. ClsMarketPrices does not relate to a database table, it is simple a class that combines the other 2 classes which may not be necessary but with my limited knowledge it is how it is working.
_ctx is a repository that links to the data context.
public MySQLRepositories(ApplicationDbContext ctx)
{
_ctx = ctx;
}
public class ClsMarket
{
[Key]
public int CompanyId { get; set; } = 0;
public string securityId { get; set; } = "";
public string companyName { get; set; } = "";
public string mic { get; set; } = "";
public string currency { get; set; } = "";
[ForeignKey("securityId")]
public virtual ICollection<ClsPrices> Prices { get; set; }
}
public class ClsMarketPrices
{
[Key]
public int CompanyId { get; set; } = 0;
public string companyName { get; set; } = "";
public string period { get; set; } = "";
public string mic { get; set; } = "";
}
public class ClsPrices
{
[Key]
public int PricesId { get; set; }
[ForeignKey("securityId")]
public string securityId { get; set; } = "";
public string mic { get; set; } = "";
public string date01 { get; set; } = "";
public virtual ClsMarket ClsMarket {get; set;}
}
I want to get a record from the 1st file joined with a record from the 2nd file where that record from the 2nd file has a date equal to or the last before the nominated date.
So we are talking about files, not a database! This is important, because this means that your local process will execute the LINQ, not a database management system. In other words: the LINQ will be IEnumerable, not IQueryable.
This is important, because as Enumerable, you will be able to define your own LINQ extension methods.
Although you supplied an enormous amount of irrelevant properties, you forgot to give us the most important things: you were talking about two files, you told us that you have two classes with a one-to-many relation, but you gave us three classes. Which ones do have the relation that you are talking about?
I think that every object of ClsMarketPrices has zero or more ClsPrices, and that every ClsPrice is one of the prices of a ClsMarketPrices, namely the ClsMarketPrices that the foreign key SecurityId (rather confusing name) refers to.
First of all, let's assume you already have procedures to read the two sequences from your files. And of course, these procedures won't read more than needed (so don't read the whole file if you will only use the first ClsMarket). I assume you already know how to do that:
IEnumerable<ClsMarketPrices> ReadMarketPrices();
IEnumerable<ClsPrices> ReadPrices();
So you've go a DateTime reviewDate. Every MarketPrice has zero or more Prices. Every Price has a DateTime property DateStamp. You want for every MarketPrice the Price that has the largest value for DateStamp that is smaller or equal to reviewDate.
If a MarketPrice doesn't have such a Prices, for instance because it doesn't have a Price at all, or all its Prices have a DateStamp larger than reviewDate, you want a value null.
You didn't say what you want if a MarketPrice has several Prices with equal largest DateStamp <= reviewDate. I assume that you don't care which one is selected.
The straighforward LINQ method would be to use GroupJoin, Where, Orderby and FirstOrDefault:
DateTime reviewDate = ...
IEnumerable<ClsMarketPrices> marketPricess = ReadMarketPrices();
IEnumerable<ClsPrices> prices = ReadPrices().Where(price => price.DateStamp <= reviewDate);
// GroupJoin marketPrices with prices:
var result = markets.GroupJoin(prices,
marketPrice => marketPrice.CompanyId, // from every MarketPrice take the primary key
price => price.CompanyId, // from every price take the foreign key to its market
// parameter resultSelector: from every market, with its zero or more matching prices
// make one new:
(marketPrice, pricesOfThisMarketPrice) => new
{
// select the marketPrice properties that you plan to use:
Id = marketPrice.CompanyId,
Name = ...
...
// from all prices of this marketPrice, take the one with the largest DateStamp
// we know there are no marketPrices with a DataStamp larger than reviewData
LatestPrice = pricesOfThisMarketPrice.OrderbyDescending(price => price.DateStamp)
.Select(price => new
{
// Select the price properties you plan to use;
Id = price.PricesId,
Date = price.DateStamp,
...
})
.FirstOrDefault(),
});
The problem is: this must be done efficiently, because you have an immense amount of Markets and MarketPrices.
Althoug we already limited the amount of prices to sort by removing the prices that are after reviewDate, it is still a waste to order all Dates if you will only be using the first one.
We can optimize this, by using Aggregate for pricesOfThisMarketPrice. This will assert that pricesOfThisMarketPrice will be enumerated only once.
Side remarks: Aggregate only works on IEnumerable, not on IQueryable, so it won't work on a database. Furthermore, pricesOfThisMarketPrice might be an empty sequence; we have to take care of that.
LatestPrice = pricesOfThisMarketPrice.Any() ?
pricesOfThisMarketPrice.Aggregate(
// select the one with the largest value of DateStamp:
(latestPrice, nextPrice) => nextPrice.DateStamp >= latesPrice.DateStamp) ? nextPrice : latestPrice)
// do not do the aggregate if there are no prices at all:
: null,
Although this Aggregate is more efficient than OrderBy, your second sequence will still be enumerated more than once. See the source code of Enumerable.GroupJoin.
If you really want to enumerate your second source once, and limit the number of enumerations of the first source, consider to create an extension method. This way you can use it as any LINQ method. If you are not familiar with extension methods, see extension methods demystified.
You can create an extension method for your ClsPrices and ClsPrice, however, if you think you will need to "find the largest element that belongs to another element" more often, why not create a generic method, just like LINQ does.
Below I create the most extensive extension method, one with a resultSelector and equalityComparers. If you will use standard equality, consider to add an extension method without these comparers and let this extension method call the other extension method with null value for the comparers.
For examples about the overloads with and without equality comparers see several LINQ methods, like ToDictionary: there is a method without a comparer and one with a comparer. This first one calls the second one with null value for comparer.
I will use baby steps, so you can understand what happens.
This can slightly be optimized.
The most important thing is that you will enumerate your largest collection only once.
IEnumerable<TResult> TakeLargestItem<T1, T2, TKey, Tproperty, TResult>(
this IEnumerable<T1> t1Sequence,
IEnumerable<T2> t2Sequence,
// Select primary and foreign key:
Func<T1, TKey> t1KeySelector,
Func<T2, TKey> t2KeySelector,
// Select the property of T2 of which you want the largest element
Func<T2, TProperty> propertySelector,
// The largest element must be <= propertyLimit:
TProperty propertyLimit,
// From T1 and the largest T2 create one TResult
Func<T1, T2, TResult> resultSelector,
// equality comparer to compare equality of primary and foreign key
IEqualityComparer<TKey> keyComparer,
// comparer to find the largest property value
IComparer<TProperty> propertyComparer)
{
// TODO: invent a property method name
// TODO: decide what to do if null input
// if no comparers provided, use the default comparers:
if (keyComparer == null) keyComparer = EqualityComparer<TKey>.Default;
if (propertyComparer == null) propertyComparer = Comparer<TProperty>.Default;
// TODO: implement
}
The implementation is straightforward:
put all T1 in a dictionary t1Key as key, {T1, T2} as value, keyComparer as comparer
then enumerate T2 only once.
check if the property <= propertyLimit,
if so, search in the dictionary for the {T1, T2} combination with the same key
check if the current t2Item is larger than the T2 in the {T1, T2} combination
if so: replace
We need an internal class:
class DictionaryValue
{
public T1 T1 {get; set;}
public T2 T2 {get; set;}
}
The code:
IDictionary<TKey, DictionaryValue> t1Dict = t1Sequence.ToDictionary(
t1 -> t1KeySelector(t1),
t1 => new DictionaryValue {T1 = t1, T2 = (T2)null },
keyComparer);
The enumeration of t2Sequence:
foreach (T2 t2 in t2Sequence)
{
// check if the property is <= propertyLimit
TProperty property = propertySelector(t2);
if (propertyComparer.Compare(property, propertyLimit) < 0)
{
// find the T1 that belongs to this T2:
TKey key = keySelector(t2);
if (t1Dict.TryGetValue(key, out DictionaryValue largestValue))
{
// there is a DictionaryValue with the same key
// is it null? then t2 is the largest
// if not null: get the property of the largest value and use the
// propertyComparer to see which one of them is the largest
if (largestValue.T2 == null)
{
largestValue.T2 = t2;
}
else
{
TProperty largestProperty = propertySelector(largestValue.T2);
if (propertyComparer.Compare(property, largestProperty) > 0)
{
// t2 has a larger property than the largestValue: replace
largestValue.T2 = t2,
}
}
}
}
}
So for every t1, we have found the largest t2 that has a property <= propertyLimit.
Use the resultSelector to create the results.
IEnumerable<TResult> result = t1Dict.Values.Select(
t1WithLargestT2 => resultSelector(t1WithLargestT2.T1, t1WithLargestT2.T2));
return result;

Entity Framework Serializable Transaction Deadlock

I have to insert a row into the database but the problem is that the primary key is generated based on the total counts of rows.
E.g. if the db has 25601 rows, the ID of the newly inserted record would be CT25602.
I want to use transactions for primary key collisions.
Here is the code I wrote.
public void CreateContact(ContactViewModel input)
{
var transactionScopeOptions = new TransactionOptions
{
IsolationLevel = IsolationLevel.Serializable,
Timeout = TimeSpan.MaxValue
};
using (TransactionScope transaction = new TransactionScope(TransactionScopeOption.Required, transactionScopeOptions))
{
var contactNo = GenerateIdentity();
var contact = MapContactFields(new NavContact { No_ = contactNo }, input);
_db.Contacts.InsertOnSubmit(contact);
_db.SubmitChanges();
transaction.Complete();
}
}
This code gives me deadlocks if two persons are trying to insert a contact in a small timespan.
Any suggestions ? Thank you
Yes, the scenario you described is very likely to deadlock. I would recommend using a sequence instead. If not, then one solution is to acquire an exclusive app lock in the transaction, before scannig for the next identity. See sp_getapplock.

Having difficulty with specific Future<T> query and collections with nHibernate.

For the sake of example, I am removing non-queried and non-essential data just to figure out how to do the initial query here.
I have a model structure like this.
class Path {
Guid Id { get; protected set; }
IList<Step> Steps { get; set; }
void AddStep(Step entity) {
// write up bidirectional association
}
}
class Step {
Guid Id { get; protected set; }
Path Path { get; set; }
// other data irreleveent
}
Now assuming 50000 steps, each with 5000 steps... I do realize I don't want to return all of them at once. But putting a limit on my query fetch isn't my real problem.
Here is the exact query I am attempting to use. I am getting the exception..
NHibernate.QueryException : duplicate alias: lpStep
----> System.ArgumentException : An item with the same key has already been added.
I'm not entirely sure how to handle this scenario. if I use a flat out Fetch on the Path query, I get Select+N errors from the NHibernate Profiler.
I do have batching enabled - but as far as I am aware, that only really applies to inserts, not retrievals. But in any case I am getting back these errors and not sure how to handle it. Any ideas?
using (var Transaction = Session.BeginTransaction()) {
Path lpPath = null;
Step lpStep = null;
var lpPaths = Session.QueryOver<Path>(() => lpPath)
.Take(50)
.Future<Path>();
var lpSteps = Session.QueryOver<Step>(() => lpStep)
.JoinAlias(() => lpPath.Steps, () => lpStep)
.Where(o => o.Path.Id == lpPath.Id)
.Take(12)
.Future<Step>();
Transaction.Commit();
foreach (var path in lpPaths) {
Console.WriteLine("{0} fetched {1} Steps",
path.Id, path.Steps.Count);
}
}
I basically want to say ..
Select (50) Paths, also, as a separate select but part of the same trip, Select the first (12) Steps that belong the previously selected Paths.
But if I use a flat out join, I get 110 rows, whereas I expect to have 2 tables, 1 of 50 rows, 1 of 600 rows.
Can someone explain to me what I am doing wrong?
mind you, I can do some minor alterations and the query runs, but it isn't 'optimized'. I can get the data I want, but it takes multiple trips and lazy loading. I can optimize the actual Path selection easily enough but it is those blasted Steps. If I just take a restrictive where clause out of the lpSteps query, it just returns the first 12 steps, not returning 12 steps for each query done.
I've looked at some of the other stack overflow posts on Future<T> and found them to look a lot like this. So I don't understand why it isn't working. I suspect that what is going on is this..
lpPaths runs.
lpSteps tries to run, first one succeeds.
lpSteps then tries to run again, finds it cannot redefine lpPaths.
Apocolypse
I'm really hoping someone smarter than me can enlighten me on the absolute most optimal way to write this.
i cant really understand what your use case is. why do you only need the first 12 Steps of each Path? What about batches of Steps to process
IList<Guid> pathIds;
while ((pathIds = QueryOver.For<Path>()
.Where(...)
.Projection(path => path.Id)
.SetmaxResults(100)).Count > 0)
{
int batch = 0;
const int batchsize = 600;
IList<Step> steps;
while ((steps = Session.QueryOver<Step>()
.Where(step => step.Path.Id).In(pathIds)
.Where(step => step. ...)
.SetFirstResult(batch * batchsize)
.Take(batchsize)
.List<Step>()).Count > 0)
{
DoSomething(steps);
batch++;
}
}

error, string or binary data would be truncated when trying to insert

I am running data.bat file with the following lines:
Rem Tis batch file will populate tables
cd\program files\Microsoft SQL Server\MSSQL
osql -U sa -P Password -d MyBusiness -i c:\data.sql
The contents of the data.sql file is:
insert Customers
(CustomerID, CompanyName, Phone)
Values('101','Southwinds','19126602729')
There are 8 more similar lines for adding records.
When I run this with start > run > cmd > c:\data.bat, I get this error message:
1>2>3>4>5>....<1 row affected>
Msg 8152, Level 16, State 4, Server SP1001, Line 1
string or binary data would be truncated.
<1 row affected>
<1 row affected>
<1 row affected>
<1 row affected>
<1 row affected>
<1 row affected>
Also, I am a newbie obviously, but what do Level #, and state # mean, and how do I look up error messages such as the one above: 8152?
From #gmmastros's answer
Whenever you see the message....
string or binary data would be truncated
Think to yourself... The field is NOT big enough to hold my data.
Check the table structure for the customers table. I think you'll find that the length of one or more fields is NOT big enough to hold the data you are trying to insert. For example, if the Phone field is a varchar(8) field, and you try to put 11 characters in to it, you will get this error.
I had this issue although data length was shorter than the field length.
It turned out that the problem was having another log table (for audit trail), filled by a trigger on the main table, where the column size also had to be changed.
In one of the INSERT statements you are attempting to insert a too long string into a string (varchar or nvarchar) column.
If it's not obvious which INSERT is the offender by a mere look at the script, you could count the <1 row affected> lines that occur before the error message. The obtained number plus one gives you the statement number. In your case it seems to be the second INSERT that produces the error.
Just want to contribute with additional information: I had the same issue and it was because of the field wasn't big enough for the incoming data and this thread helped me to solve it (the top answer clarifies it all).
BUT it is very important to know what are the possible reasons that may cause it.
In my case i was creating the table with a field like this:
Select '' as Period, * From Transactions Into #NewTable
Therefore the field "Period" had a length of Zero and causing the Insert operations to fail. I changed it to "XXXXXX" that is the length of the incoming data and it now worked properly (because field now had a lentgh of 6).
I hope this help anyone with same issue :)
Some of your data cannot fit into your database column (small). It is not easy to find what is wrong. If you use C# and Linq2Sql, you can list the field which would be truncated:
First create helper class:
public class SqlTruncationExceptionWithDetails : ArgumentOutOfRangeException
{
public SqlTruncationExceptionWithDetails(System.Data.SqlClient.SqlException inner, DataContext context)
: base(inner.Message + " " + GetSqlTruncationExceptionWithDetailsString(context))
{
}
/// <summary>
/// PArt of code from following link
/// http://stackoverflow.com/questions/3666954/string-or-binary-data-would-be-truncated-linq-exception-cant-find-which-fiel
/// </summary>
/// <param name="context"></param>
/// <returns></returns>
static string GetSqlTruncationExceptionWithDetailsString(DataContext context)
{
StringBuilder sb = new StringBuilder();
foreach (object update in context.GetChangeSet().Updates)
{
FindLongStrings(update, sb);
}
foreach (object insert in context.GetChangeSet().Inserts)
{
FindLongStrings(insert, sb);
}
return sb.ToString();
}
public static void FindLongStrings(object testObject, StringBuilder sb)
{
foreach (var propInfo in testObject.GetType().GetProperties())
{
foreach (System.Data.Linq.Mapping.ColumnAttribute attribute in propInfo.GetCustomAttributes(typeof(System.Data.Linq.Mapping.ColumnAttribute), true))
{
if (attribute.DbType.ToLower().Contains("varchar"))
{
string dbType = attribute.DbType.ToLower();
int numberStartIndex = dbType.IndexOf("varchar(") + 8;
int numberEndIndex = dbType.IndexOf(")", numberStartIndex);
string lengthString = dbType.Substring(numberStartIndex, (numberEndIndex - numberStartIndex));
int maxLength = 0;
int.TryParse(lengthString, out maxLength);
string currentValue = (string)propInfo.GetValue(testObject, null);
if (!string.IsNullOrEmpty(currentValue) && maxLength != 0 && currentValue.Length > maxLength)
{
//string is too long
sb.AppendLine(testObject.GetType().Name + "." + propInfo.Name + " " + currentValue + " Max: " + maxLength);
}
}
}
}
}
}
Then prepare the wrapper for SubmitChanges:
public static class DataContextExtensions
{
public static void SubmitChangesWithDetailException(this DataContext dataContext)
{
//http://stackoverflow.com/questions/3666954/string-or-binary-data-would-be-truncated-linq-exception-cant-find-which-fiel
try
{
//this can failed on data truncation
dataContext.SubmitChanges();
}
catch (SqlException sqlException) //when (sqlException.Message == "String or binary data would be truncated.")
{
if (sqlException.Message == "String or binary data would be truncated.") //only for EN windows - if you are running different window language, invoke the sqlException.getMessage on thread with EN culture
throw new SqlTruncationExceptionWithDetails(sqlException, dataContext);
else
throw;
}
}
}
Prepare global exception handler and log truncation details:
protected void Application_Error(object sender, EventArgs e)
{
Exception ex = Server.GetLastError();
string message = ex.Message;
//TODO - log to file
}
Finally use the code:
Datamodel.SubmitChangesWithDetailException();
Another situation in which you can get this error is the following:
I had the same error and the reason was that in an INSERT statement that received data from an UNION, the order of the columns was different from the original table. If you change the order in #table3 to a, b, c, you will fix the error.
select a, b, c into #table1
from #table0
insert into #table1
select a, b, c from #table2
union
select a, c, b from #table3
on sql server you can use SET ANSI_WARNINGS OFF like this:
using (SqlConnection conn = new SqlConnection("Data Source=XRAYGOAT\\SQLEXPRESS;Initial Catalog='Healthy Care';Integrated Security=True"))
{
conn.Open();
using (var trans = conn.BeginTransaction())
{
try
{
using cmd = new SqlCommand("", conn, trans))
{
cmd.CommandText = "SET ANSI_WARNINGS OFF";
cmd.ExecuteNonQuery();
cmd.CommandText = "YOUR INSERT HERE";
cmd.ExecuteNonQuery();
cmd.Parameters.Clear();
cmd.CommandText = "SET ANSI_WARNINGS ON";
cmd.ExecuteNonQuery();
trans.Commit();
}
}
catch (Exception)
{
trans.Rollback();
}
}
conn.Close();
}
I had the same issue. The length of my column was too short.
What you can do is either increase the length or shorten the text you want to put in the database.
Also had this problem occurring on the web application surface.
Eventually found out that the same error message comes from the SQL update statement in the specific table.
Finally then figured out that the column definition in the relating history table(s) did not map the original table column length of nvarchar types in some specific cases.
I had the same problem, even after increasing the size of the problematic columns in the table.
tl;dr: The length of the matching columns in corresponding Table Types may also need to be increased.
In my case, the error was coming from the Data Export service in Microsoft Dynamics CRM, which allows CRM data to be synced to an SQL Server DB or Azure SQL DB.
After a lengthy investigation, I concluded that the Data Export service must be using Table-Valued Parameters:
You can use table-valued parameters to send multiple rows of data to a Transact-SQL statement or a routine, such as a stored procedure or function, without creating a temporary table or many parameters.
As you can see in the documentation above, Table Types are used to create the data ingestion procedure:
CREATE TYPE LocationTableType AS TABLE (...);
CREATE PROCEDURE dbo.usp_InsertProductionLocation
#TVP LocationTableType READONLY
Unfortunately, there is no way to alter a Table Type, so it has to be dropped & recreated entirely. Since my table has over 300 fields (😱), I created a query to facilitate the creation of the corresponding Table Type based on the table's columns definition (just replace [table_name] with your table's name):
SELECT 'CREATE TYPE [table_name]Type AS TABLE (' + STRING_AGG(CAST(field AS VARCHAR(max)), ',' + CHAR(10)) + ');' AS create_type
FROM (
SELECT TOP 5000 COLUMN_NAME + ' ' + DATA_TYPE
+ IIF(CHARACTER_MAXIMUM_LENGTH IS NULL, '', CONCAT('(', IIF(CHARACTER_MAXIMUM_LENGTH = -1, 'max', CONCAT(CHARACTER_MAXIMUM_LENGTH,'')), ')'))
+ IIF(DATA_TYPE = 'decimal', CONCAT('(', NUMERIC_PRECISION, ',', NUMERIC_SCALE, ')'), '')
AS field
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = '[table_name]'
ORDER BY ORDINAL_POSITION) AS T;
After updating the Table Type, the Data Export service started functioning properly once again! :)
When I tried to execute my stored procedure I had the same problem because the size of the column that I need to add some data is shorter than the data I want to add.
You can increase the size of the column data type or reduce the length of your data.
A 2016/2017 update will show you the bad value and column.
A new trace flag will swap the old error for a new 2628 error and will print out the column and offending value. Traceflag 460 is available in the latest cumulative update for 2016 and 2017:
https://support.microsoft.com/en-sg/help/4468101/optional-replacement-for-string-or-binary-data-would-be-truncated
Just make sure that after you've installed the CU that you enable the trace flag, either globally/permanently on the server:
...or with DBCC TRACEON:
https://learn.microsoft.com/en-us/sql/t-sql/database-console-commands/dbcc-traceon-trace-flags-transact-sql?view=sql-server-ver15
Another situation, in which this error may occur is in
SQL Server Management Studio. If you have "text" or "ntext" fields in your table,
no matter what kind of field you are updating (for example bit or integer).
Seems that the Studio does not load entire "ntext" fields and also updates ALL fields instead of the modified one.
To solve the problem, exclude "text" or "ntext" fields from the query in Management Studio
This Error Comes only When any of your field length is greater than the field length specified in sql server database table structure.
To overcome this issue you have to reduce the length of the field Value .
Or to increase the length of database table field .
If someone is encountering this error in a C# application, I have created a simple way of finding offending fields by:
Getting the column width of all the columns of a table where we're trying to make this insert/ update. (I'm getting this info directly from the database.)
Comparing the column widths to the width of the values we're trying to insert/ update.
Assumptions/ Limitations:
The column names of the table in the database match with the C# entity fields. For eg: If you have a column like this in database:
You need to have your Entity with the same column name:
public class SomeTable
{
// Other fields
public string SourceData { get; set; }
}
You're inserting/ updating 1 entity at a time. It'll be clearer in the demo code below. (If you're doing bulk inserts/ updates, you might want to either modify it or use some other solution.)
Step 1:
Get the column width of all the columns directly from the database:
// For this, I took help from Microsoft docs website:
// https://learn.microsoft.com/en-us/dotnet/api/system.data.sqlclient.sqlconnection.getschema?view=netframework-4.7.2#System_Data_SqlClient_SqlConnection_GetSchema_System_String_System_String___
private static Dictionary<string, int> GetColumnSizesOfTableFromDatabase(string tableName, string connectionString)
{
var columnSizes = new Dictionary<string, int>();
using (var connection = new SqlConnection(connectionString))
{
// Connect to the database then retrieve the schema information.
connection.Open();
// You can specify the Catalog, Schema, Table Name, Column Name to get the specified column(s).
// You can use four restrictions for Column, so you should create a 4 members array.
String[] columnRestrictions = new String[4];
// For the array, 0-member represents Catalog; 1-member represents Schema;
// 2-member represents Table Name; 3-member represents Column Name.
// Now we specify the Table_Name and Column_Name of the columns what we want to get schema information.
columnRestrictions[2] = tableName;
DataTable allColumnsSchemaTable = connection.GetSchema("Columns", columnRestrictions);
foreach (DataRow row in allColumnsSchemaTable.Rows)
{
var columnName = row.Field<string>("COLUMN_NAME");
//var dataType = row.Field<string>("DATA_TYPE");
var characterMaxLength = row.Field<int?>("CHARACTER_MAXIMUM_LENGTH");
// I'm only capturing columns whose Datatype is "varchar" or "char", i.e. their CHARACTER_MAXIMUM_LENGTH won't be null.
if(characterMaxLength != null)
{
columnSizes.Add(columnName, characterMaxLength.Value);
}
}
connection.Close();
}
return columnSizes;
}
Step 2:
Compare the column widths with the width of the values we're trying to insert/ update:
public static Dictionary<string, string> FindLongBinaryOrStringFields<T>(T entity, string connectionString)
{
var tableName = typeof(T).Name;
Dictionary<string, string> longFields = new Dictionary<string, string>();
var objectProperties = GetProperties(entity);
//var fieldNames = objectProperties.Select(p => p.Name).ToList();
var actualDatabaseColumnSizes = GetColumnSizesOfTableFromDatabase(tableName, connectionString);
foreach (var dbColumn in actualDatabaseColumnSizes)
{
var maxLengthOfThisColumn = dbColumn.Value;
var currentValueOfThisField = objectProperties.Where(f => f.Name == dbColumn.Key).First()?.GetValue(entity, null)?.ToString();
if (!string.IsNullOrEmpty(currentValueOfThisField) && currentValueOfThisField.Length > maxLengthOfThisColumn)
{
longFields.Add(dbColumn.Key, $"'{dbColumn.Key}' column cannot take the value of '{currentValueOfThisField}' because the max length it can take is {maxLengthOfThisColumn}.");
}
}
return longFields;
}
public static List<PropertyInfo> GetProperties<T>(T entity)
{
//The DeclaredOnly flag makes sure you only get properties of the object, not from the classes it derives from.
var properties = entity.GetType()
.GetProperties(System.Reflection.BindingFlags.Public
| System.Reflection.BindingFlags.Instance
| System.Reflection.BindingFlags.DeclaredOnly)
.ToList();
return properties;
}
Demo:
Let's say we're trying to insert someTableEntity of SomeTable class that is modeled in our app like so:
public class SomeTable
{
[Key]
public long TicketID { get; set; }
public string SourceData { get; set; }
}
And it's inside our SomeDbContext like so:
public class SomeDbContext : DbContext
{
public DbSet<SomeTable> SomeTables { get; set; }
}
This table in Db has SourceData field as varchar(16) like so:
Now we'll try to insert value that is longer than 16 characters into this field and capture this information:
public void SaveSomeTableEntity()
{
var connectionString = "server=SERVER_NAME;database=DB_NAME;User ID=SOME_ID;Password=SOME_PASSWORD;Connection Timeout=200";
using (var context = new SomeDbContext(connectionString))
{
var someTableEntity = new SomeTable()
{
SourceData = "Blah-Blah-Blah-Blah-Blah-Blah"
};
context.SomeTables.Add(someTableEntity);
try
{
context.SaveChanges();
}
catch (Exception ex)
{
if (ex.GetBaseException().Message == "String or binary data would be truncated.\r\nThe statement has been terminated.")
{
var badFieldsReport = "";
List<string> badFields = new List<string>();
// YOU GOT YOUR FIELDS RIGHT HERE:
var longFields = FindLongBinaryOrStringFields(someTableEntity, connectionString);
foreach (var longField in longFields)
{
badFields.Add(longField.Key);
badFieldsReport += longField.Value + "\n";
}
}
else
throw;
}
}
}
The badFieldsReport will have this value:
'SourceData' column cannot take the value of
'Blah-Blah-Blah-Blah-Blah-Blah' because the max length it can take is
16.
Kevin Pope's comment under the accepted answer was what I needed.
The problem, in my case, was that I had triggers defined on my table that would insert update/insert transactions into an audit table, but the audit table had a data type mismatch where a column with VARCHAR(MAX) in the original table was stored as VARCHAR(1) in the audit table, so my triggers were failing when I would insert anything greater than VARCHAR(1) in the original table column and I would get this error message.
I used a different tactic, fields that are allocated 8K in some places. Here only about 50/100 are used.
declare #NVPN_list as table
nvpn varchar(50)
,nvpn_revision varchar(5)
,nvpn_iteration INT
,mpn_lifecycle varchar(30)
,mfr varchar(100)
,mpn varchar(50)
,mpn_revision varchar(5)
,mpn_iteration INT
-- ...
) INSERT INTO #NVPN_LIST
SELECT left(nvpn ,50) as nvpn
,left(nvpn_revision ,10) as nvpn_revision
,nvpn_iteration
,left(mpn_lifecycle ,30)
,left(mfr ,100)
,left(mpn ,50)
,left(mpn_revision ,5)
,mpn_iteration
,left(mfr_order_num ,50)
FROM [DASHBOARD].[dbo].[mpnAttributes] (NOLOCK) mpna
I wanted speed, since I have 1M total records, and load 28K of them.
This error may be due to less field size than your entered data.
For e.g. if you have data type nvarchar(7) and if your value is 'aaaaddddf' then error is shown as:
string or binary data would be truncated
You simply can't beat SQL Server on this.
You can insert into a new table like this:
select foo, bar
into tmp_new_table_to_dispose_later
from my_table
and compare the table definition with the real table you want to insert the data into.
Sometime it's helpful sometimes it's not.
If you try inserting in the final/real table from that temporary table it may just work (due to data conversion working differently than SSMS for example).
Another alternative is to insert the data in chunks, instead of inserting everything immediately you insert with top 1000 and you repeat the process, till you find a chunk with an error. At least you have better visibility on what's not fitting into the table.