I have recently had performance problem with Nhibernate generated SQL as described in
Nhibernate generate plain sql query instead of execution statement
I also found a link describing similar experience from the one and only Jeff Atwood at
http://legeronline.blogspot.ca/2009/03/evils-of-slow-paramaterized-query-plans.html
Does anyone know if there is anyway to add an "Optimize Uknown" option to Nhibernate?
This example is a little more verbose:
public class OptionInterceptor: EmptyInterceptor
{
public override SqlString OnPrepareStatement(SqlString sql)
{
var parameters = sql.GetParameters();
var paramCount = parameters.Count();
if (paramCount == 0)
return sql;
string optionString = " OPTION (OPTIMIZE FOR (";
for (var i = 0; i < paramCount; i++)
{
var comma = i > 0 ? "," : string.Empty;
optionString = optionString + comma + "#p" + i + " UNKNOWN";
}
optionString = optionString + "))";
var builder = new SqlStringBuilder(sql);
builder.Add(optionString);
return builder.ToSqlString();
}
}
Then in your sessionfactory/initializer:
configuration.ExposeConfiguration(x =>
{
x.SetInterceptor(new OptionInterceptor());
});
you could look into extending the MsSqlDialect or
implement IConnectionProvider to inject your own commandwrapper which adds the hint when ExecuteReader() is called
Related
How do I run a raw query in ASP.NET core to return the row count from a table?
Currently, I am doing this and the returned result is -1. I think the return result is based on the number of records affected.
int numberOfRows = await
appDbContext.Database.ExecuteSqlInterpolatedAsync(
$"SELECT CODE FROM [samaster] WHERE CODE={productBrandCode} AND WAREHOUSE={warehouse} ");
Any idea on how to get the count back to numberOfRows variable will be appreciated.
NOTE: The above table is not a model so I need to run a raw query.
Thanks
It is currently not possible to get the query result when using ExecuteSqlInterpolatedAsync. The same applies to any additional LINQ Statements.
You can, however, use the underlying ADO.net Provider:
public IList<IDictionary<string, dynamic>> SelectDynamic(string table)
{
using (var command = Database.GetDbConnection().CreateCommand())
{
command.CommandText = $"SELECT * FROM [{table}]";
command.CommandType = CommandType.Text;
Database.OpenConnection();
using (var result = command.ExecuteReader())
{
var entities = new List<IDictionary<string, dynamic>>();
while (result.Read())
{
var dict = new Dictionary<string, dynamic>();
for (int i = 0; i < result.FieldCount; i++)
{
dict.Add(result.GetName(i), result.GetValue(i));
}
entities.Add(dict);
}
return entities;
}
}
}
Add this to your DbContext Class and Call it with:
using (var context = new MyDbContext()) // Or get it with DI, depends on your application
{
var count = context.SelectDynamic("samaster").Where(d => d["CODE"] == productBrandCode && d["WAREHOUSE"] == warehouse).Count();
}
Beware, however, that this is an expensive operation if you have a lot of rows in your table!
An alternative approach to only fetch the relevant results would be to replace
command.CommandText = $"SELECT * FROM [{table}]";
with
command.CommandText = $"SELECT CODE FROM [samaster] WHERE CODE={productBrandCode} AND WAREHOUSE={warehouse}";
and pass the parameters as function parameters.
public IList<IDictionary<string, dynamic>> SelectDynamic(string productBrandCode, string warehouse)
{...
Also make sure to escape all parameters if they are in any way submitted by user input to prevent SQL Injection Attacks!
There are two common approaches :
A.
int numberOfRows = await appDbContext.Database.ExecuteSqlInterpolatedAsync($"SELECT CODE FROM [samaster] WHERE CODE={productBrandCode} AND WAREHOUSE={warehouse} ").Count();
B.
int numberOfRows = await appDbContext.Database.ExecuteSqlInterpolatedAsync($"SELECT count(*) FROM [samaster] WHERE CODE={productBrandCode} AND WAREHOUSE={warehouse} ").First();
Since nobody gave me the correct answer. I end up using the following.
public async Task<bool> IsAValidProduct(string productBrandCode)
{
int count = 0;
await using DbCommand command = appDbContext.Database.GetDbConnection().CreateCommand();
command.CommandText =
"SELECT COUNT(CODE) FROM [samaster] WHERE CODE=#productBrandCode AND WAREHOUSE=#warehouse ";
command.CommandType = CommandType.Text;
command.Parameters.Add(new SqlParameter("#productBrandCode", SqlDbType.VarChar)
{Value = productBrandCode});
command.Parameters.Add(new SqlParameter("#warehouse", SqlDbType.VarChar)
{Value = warehouse});
await appDbContext.Database.OpenConnectionAsync();
count = (int) await command.ExecuteScalarAsync();
await appDbContext.Database.CloseConnectionAsync();
return count == 1;
}
int c= dbObj.Database.ExecuteSqlRaw(sql); //User this code
this simple code works fine and allows to add a BeginSample/EndSample call around each Update/LateUpdate/FixedUpdate function. However it doesn't take in consideration early return instructions, for example as result of a condition. Do you know how to write a similar function that take in considerations early returns so that the EndSample call will be executed under every circumstance?
Note that I am not a Cecil expert, I am just learning now. It appears to me that Cecil automatically updates the operations that returns early after calling InsertBefore and similar functions. So if a BR opcode was previously jumping to a specific instruction address, the address will be updated after the insertions in order to jump to the original instruction. This is OK in most of the cases, but in my case it means that an if statement would skip the last inserted operation as the BR operation would still point directly to the final Ret instruction. Note that Update, LateUpdate and FixedUpdate are all void functions.
foreach (var method in type.Methods)
{
if ((method.Name == "Update" || method.Name == "LateUpdate" || method.Name == "FixedUpdate") &&
method.HasParameters == false)
{
var beginMethod =
module.ImportReference(typeof (Profiler).GetMethod("BeginSample",
new[] {typeof (string)}));
var endMethod =
module.ImportReference(typeof (Profiler).GetMethod("EndSample",
BindingFlags.Static |
BindingFlags.Public));
Debug.Log(method.Name + " method found in class: " + type.Name);
var ilProcessor = method.Body.GetILProcessor();
var first = method.Body.Instructions[0];
ilProcessor.InsertBefore(first,
Instruction.Create(OpCodes.Ldstr,
type.FullName + "." + method.Name));
ilProcessor.InsertBefore(first, Instruction.Create(OpCodes.Call, beginMethod));
var lastRet = method.Body.Instructions[method.Body.Instructions.Count - 1];
ilProcessor.InsertBefore(lastRet, Instruction.Create(OpCodes.Call, endMethod));
changed = true;
}
}
as a Bonus, if you can explain to me the difference between Emit and Append a newly created instruction with the same operand. does Append execute an Emit under the hood or does something more?
I may have found the solution, at least apparently it works. I followed the code used to solve a similar problem from here:
https://groups.google.com/forum/#!msg/mono-cecil/nE6JBjvEFCQ/MqV6tgDCB4AJ
I adapted it for my purposes and it seemed to work, although I may find out other issues. This is the complete code:
static bool ProcessAssembly(AssemblyDefinition assembly)
{
var changed = false;
var moduleG = assembly.MainModule;
var attributeConstructor =
moduleG.ImportReference(
typeof(RamjetProfilerPostProcessedAssemblyAttribute).GetConstructor(Type.EmptyTypes));
var attribute = new CustomAttribute(attributeConstructor);
var ramjet = moduleG.ImportReference(typeof(RamjetProfilerPostProcessedAssemblyAttribute));
if (assembly.HasCustomAttributes)
{
var attributes = assembly.CustomAttributes;
foreach (var attr in attributes)
{
if (attr.AttributeType.FullName == ramjet.FullName)
{
Debug.LogWarning("<color=yellow>Skipping already-patched assembly:</color> " + assembly.Name);
return false;
}
}
}
assembly.CustomAttributes.Add(attribute);
foreach (var module in assembly.Modules)
{
foreach (var type in module.Types)
{
// Skip any classes related to the RamjetProfiler
if (type.Name.Contains("AssemblyPostProcessor") || type.Name.Contains("RamjetProfiler"))
{
// Todo: use actual type equals, not string matching
Debug.Log("Skipping self class : " + type.Name);
continue;
}
if (type.BaseType != null && type.BaseType.FullName.Contains("UnityEngine.MonoBehaviour"))
{
foreach (var method in type.Methods)
{
if ((method.Name == "Update" || method.Name == "LateUpdate" || method.Name == "FixedUpdate") &&
method.HasParameters == false)
{
var beginMethod =
module.ImportReference(typeof(Profiler).GetMethod("BeginSample",
new[] { typeof(string) }));
var endMethod =
module.ImportReference(typeof(Profiler).GetMethod("EndSample",
BindingFlags.Static |
BindingFlags.Public));
Debug.Log(method.Name + " method found in class: " + type.Name);
var ilProcessor = method.Body.GetILProcessor();
var first = method.Body.Instructions[0];
ilProcessor.InsertBefore(first,
Instruction.Create(OpCodes.Ldstr,
type.FullName + "." + method.Name));
ilProcessor.InsertBefore(first, Instruction.Create(OpCodes.Call, beginMethod));
var lastcall = Instruction.Create(OpCodes.Call, endMethod);
FixReturns(method, lastcall);
changed = true;
}
}
}
}
}
return changed;
}
static void FixReturns(MethodDefinition med, Instruction lastcall)
{
MethodBody body = med.Body;
var instructions = body.Instructions;
Instruction formallyLastInstruction = instructions[instructions.Count - 1];
Instruction lastLeaveInstruction = null;
var lastRet = Instruction.Create(OpCodes.Ret);
instructions.Add(lastcall);
instructions.Add(lastRet);
for (var index = 0; index < instructions.Count - 1; index++)
{
var instruction = instructions[index];
if (instruction.OpCode == OpCodes.Ret)
{
Instruction leaveInstruction = Instruction.Create(OpCodes.Leave, lastcall);
if (instruction == formallyLastInstruction)
{
lastLeaveInstruction = leaveInstruction;
}
instructions[index] = leaveInstruction;
}
}
FixBranchTargets(lastLeaveInstruction, formallyLastInstruction, body);
}
private static void FixBranchTargets(
Instruction lastLeaveInstruction,
Instruction formallyLastRetInstruction,
MethodBody body)
{
for (var index = 0; index < body.Instructions.Count - 2; index++)
{
var instruction = body.Instructions[index];
if (instruction.Operand != null && instruction.Operand == formallyLastRetInstruction)
{
instruction.Operand = lastLeaveInstruction;
}
}
}
basically what it does is to add a Ret instuction, but then replace all the previous Ret (usually one, why should it be more than one?) with a Leave function (don't even know what it means :) ), so that all the previous jumps remain valid. Differently than the original code, I make the Leave instruction point to the EndSample call before the last Ret
Generic Fetch XML To Get All attributes from any entity in Microsoft Dynamics CRM 2011?
if you want it to be generic, youll have to use reflection to loop through the members and build the query xml dynamically.
something like:
Type type = TypeOf(Contact);
PropertyInfo[] properties = type.GetProperties();
foreach (PropertyInfo property in properties)
{
/////here you chain the members for the xml
Console.WriteLine("Name: " + property.Name + ", Value: " + property.GetValue(obj, null));
}
To fetch details from any entity use below method.
Please read Note and Parameter Description before using this generic Fetchxml generator.
Note : fieldToQuery, operatorForCondition, fieldQueryValue Should have Array Same count to work this function and well Mappep with respective to each other to get desired result
parameter name = "entityName" = Name of Entity of which details to fetch.
parameter name = "fieldsToSearch" = What all fields you want to fetch, if Sent Null, by default all fields are fetched from entity
parameter name = "filterType" = "AND" or "OR"
parameter name = "fieldToQuery" = Array of Field name on which to query
parameter name = "operatorForCondition" = Array of operator between fieldToQuery(Field Name) and fieldQueryValue(Field Value) like "eq, like"
parameter name = "fieldQueryValue" = Array of Field Value respective to fieldToQuery (Field Name) by which to query
Function stats from here
`
public static Microsoft.Xrm.Sdk.EntityCollection RetrieveEntityDetailsByFetchXml(string entityName, string[] fieldsToSearch, string filterType, string[] fieldToQuery, string[] operatorForCondition, string[] fieldQueryValue)
{
string _fetchDetailsXml = #"<fetch version='1.0' mapping='logical' output-format='xml-platform'> <entity name='" + entityName + "'> ";
_fetchDetailsXml = GenerateFetchXMLForAttributes(fieldsToSearch, _fetchDetailsXml);
_fetchDetailsXml += "<filter type='" + filterType + "'>";
if (fieldQueryValue.Count() != fieldToQuery.Count() && fieldToQuery.Count() != operatorForCondition.Count())
{
throw new ApplicationException("FieldtoQuery and FieldQueryValue are not mapped correctly fieldToQuery : " + fieldToQuery.Count() + " fieldQueryValue : " + fieldQueryValue.Count() + ".");
}
_fetchDetailsXml = GenerateFetchXMLForConditions(fieldToQuery, operatorForCondition, fieldQueryValue, _fetchDetailsXml);
_fetchDetailsXml += "</filter> </entity> </fetch>";
Microsoft.Xrm.Sdk.EntityCollection _entityDetailsColl = CrmConnectionsManager.OrganizationServiceProxy.RetrieveMultiple(new FetchExpression(_fetchDetailsXml));
// Note : "_entityDetailsColl" cast this object to respective Entity object
if (_entityDetailsColl != null && _entityDetailsColl.Entities.Count() > 0)
{
return _entityDetailsColl;
}
return null;
} `
`
public static string GenerateFetchXMLForAttributes(string[] fieldsToSearch, string fetchXml)
{
if (fieldsToSearch != null && fieldsToSearch.Count() > 0)
{
foreach (string field in fieldsToSearch)
{
fetchXml += "<attribute name='" + field + "' />";
}
}
else
{
fetchXml += "<all-attributes/>";
}
return fetchXml;
}
public static string GenerateFetchXMLForConditions(string[]fieldToQuery,string[] operatorForCondition, string[] fieldQueryValue, string _fetchDetailsXml)
{
if (fieldToQuery != null && fieldToQuery.Count() > 0)
{
for (int count = 0; count < fieldToQuery.Count(); count++)
{
_fetchDetailsXml += "<condition attribute='" + fieldToQuery[count] + "' operator='" + operatorForCondition[count] + "' value='" + fieldQueryValue[count] + "' uiname='' uitype='' /> ";
}
}
else
{
_fetchDetailsXml += "<all-attributes/>";
}
return _fetchDetailsXml;
}
`
Retrieving “all attributes” is computationally more expensive than retrieving just those you need (also consider the IO cost). If you are looking for a way to view the attributes then write code for just those you need try this:
In the CRM web GUI:
1.Navigate to the view that displays the records in question
2.Click the Advanced Find button in the Ribbon Bar
3.Configure your “find” until it shows the records you are looking for
4.Click the Download Fetch XML button in the Ribbon Bar
5.Open the file with a text viewer (or your favorite development tool)
If you want to use this XML directly you might consider:
Use FetchXML to Construct a Query
http://msdn.microsoft.com/en-us/library/gg328117.aspx
I'm trying to implement paging within my project.
When using NHibernate's IQueryOver syntax as shown below things are working as expected.
public PagedResult<T> ExecutePagedQuery(IQueryOver<T,T> query)
{
SetPaging(query);
var results = query.Future<T>();
var count = query.ToRowCountQuery().FutureValue<int>();
return new PagedResult<T>()
{
TotalItemCount = count.Value,
PageOfResults = results.ToList()
};
}
protected virtual void SetPaging(IQueryOver<T, T> query)
{
var maxResults = PageSize;
var numberToSkip = (PageNumber - 1) * PageSize;
query.Skip(numberToSkip).Take(maxResults);
}
For some queries we are using HQL instead of the IQueryOver syntax however.
I'm wondering if there is an equivalent to "query.ToRowCountQuery().FutureValue< int >()" that can be used when querying when passing an IQuery insead of an IQueryOver to provide the row count.
public PagedResult<T> ExecutePagedQuery(IQuery query)
{
SetPaging(query);
var results = query.Future<T>();
var count = // ToRowCountQueryEquivalent?
return new PagedResult<T>()
{
TotalItemCount = count,
PageOfResults = results.ToList()
};
}
protected virtual void SetPaging(IQuery query)
{
var maxResults = PageSize;
var numberToSkip = (PageNumber - 1) * PageSize;
query.SetFirstResult(numberToSkip);
query.SetMaxResults(maxResults);
}
some crazy idea
string countQuery;
if (query.QueryString.StartsWith("SELECT", StringComparison.OrdinalIgnoreCase))
countQuery = Regex.Replace(query.QueryString, "SELECT (.*) FROM", #"SELECT Count($1) FROM", RegexOptions.IgnoreCase);
else
countQuery = "SELECT COUNT(*) " + query.QueryString;
We've since converted our HQL queries to IQueryOver allowing us to take advantage of the ToRowCountQuery() helper.
I am interfacing with a PostgreSQL database with NHibernate.
Background
I made some simple tests...it seems it's taking 2 seconds to persist 300 records.
I have a Perl program with identical functionality, but issue direct SQL instead, takes only 70% of the time.
I am not sure if this is expected. I thought C#/NHibernate would be faster or at least on par.
Questions
One of my observation is that (with show_sql turned on), the NHibernate is issuing INSERTs a few hundreds times, instead of doing bulk INSERT that take cares of multiple rows. And note I am assigning the primary key myself, not using the "native" generator.
Is that expected? Is there anyway I could make it issue bulk INSERT statement instead? It seems to me that this could be one of the area I could speed up the performance.
As stachu found out correctly: NHibernate does not have *BatchingBatcher(Factory) for PostgreSQL(Npgsql)
As stachu askes: Did anybody managed to force Nhibarnate to do batch inserts to PostgreSQL
I wrote a Batcher that doesn't use any Npgsql batching stuff, but does manipulate the SQL String "oldschool style" (INSERT INTO [..] VALUES (...),(...), ...)
using System;
using System.Collections;
using System.Data;
using System.Diagnostics;
using System.Text;
using Npgsql;
namespace NHibernate.AdoNet
{
public class PostgresClientBatchingBatcherFactory : IBatcherFactory
{
public virtual IBatcher CreateBatcher(ConnectionManager connectionManager, IInterceptor interceptor)
{
return new PostgresClientBatchingBatcher(connectionManager, interceptor);
}
}
/// <summary>
/// Summary description for PostgresClientBatchingBatcher.
/// </summary>
public class PostgresClientBatchingBatcher : AbstractBatcher
{
private int batchSize;
private int countOfCommands = 0;
private int totalExpectedRowsAffected;
private StringBuilder sbBatchCommand;
private int m_ParameterCounter;
private IDbCommand currentBatch;
public PostgresClientBatchingBatcher(ConnectionManager connectionManager, IInterceptor interceptor)
: base(connectionManager, interceptor)
{
batchSize = Factory.Settings.AdoBatchSize;
}
private string NextParam()
{
return ":p" + m_ParameterCounter++;
}
public override void AddToBatch(IExpectation expectation)
{
if(expectation.CanBeBatched && !(CurrentCommand.CommandText.StartsWith("INSERT INTO") && CurrentCommand.CommandText.Contains("VALUES")))
{
//NonBatching behavior
IDbCommand cmd = CurrentCommand;
LogCommand(CurrentCommand);
int rowCount = ExecuteNonQuery(cmd);
expectation.VerifyOutcomeNonBatched(rowCount, cmd);
currentBatch = null;
return;
}
totalExpectedRowsAffected += expectation.ExpectedRowCount;
log.Info("Adding to batch");
int len = CurrentCommand.CommandText.Length;
int idx = CurrentCommand.CommandText.IndexOf("VALUES");
int endidx = idx + "VALUES".Length + 2;
if (currentBatch == null)
{
// begin new batch.
currentBatch = new NpgsqlCommand();
sbBatchCommand = new StringBuilder();
m_ParameterCounter = 0;
string preCommand = CurrentCommand.CommandText.Substring(0, endidx);
sbBatchCommand.Append(preCommand);
}
else
{
//only append Values
sbBatchCommand.Append(", (");
}
//append values from CurrentCommand to sbBatchCommand
string values = CurrentCommand.CommandText.Substring(endidx, len - endidx - 1);
//get all values
string[] split = values.Split(',');
ArrayList paramName = new ArrayList(split.Length);
for (int i = 0; i < split.Length; i++ )
{
if (i != 0)
sbBatchCommand.Append(", ");
string param = null;
if (split[i].StartsWith(":")) //first named parameter
{
param = NextParam();
paramName.Add(param);
}
else if(split[i].StartsWith(" :")) //other named parameter
{
param = NextParam();
paramName.Add(param);
}
else if (split[i].StartsWith(" ")) //other fix parameter
{
param = split[i].Substring(1, split[i].Length-1);
}
else
{
param = split[i]; //first fix parameter
}
sbBatchCommand.Append(param);
}
sbBatchCommand.Append(")");
//rename & copy parameters from CurrentCommand to currentBatch
int iParam = 0;
foreach (NpgsqlParameter param in CurrentCommand.Parameters)
{
param.ParameterName = (string)paramName[iParam++];
NpgsqlParameter newParam = /*Clone()*/new NpgsqlParameter(param.ParameterName, param.NpgsqlDbType, param.Size, param.SourceColumn, param.Direction, param.IsNullable, param.Precision, param.Scale, param.SourceVersion, param.Value);
currentBatch.Parameters.Add(newParam);
}
countOfCommands++;
//check for flush
if (countOfCommands >= batchSize)
{
DoExecuteBatch(currentBatch);
}
}
protected override void DoExecuteBatch(IDbCommand ps)
{
if (currentBatch != null)
{
//Batch command now needs its terminator
sbBatchCommand.Append(";");
countOfCommands = 0;
log.Info("Executing batch");
CheckReaders();
//set prepared batchCommandText
string commandText = sbBatchCommand.ToString();
currentBatch.CommandText = commandText;
LogCommand(currentBatch);
Prepare(currentBatch);
int rowsAffected = 0;
try
{
rowsAffected = currentBatch.ExecuteNonQuery();
}
catch (Exception e)
{
if(Debugger.IsAttached)
Debugger.Break();
throw;
}
Expectations.VerifyOutcomeBatched(totalExpectedRowsAffected, rowsAffected);
totalExpectedRowsAffected = 0;
currentBatch = null;
sbBatchCommand = null;
m_ParameterCounter = 0;
}
}
protected override int CountOfStatementsInCurrentBatch
{
get { return countOfCommands; }
}
public override int BatchSize
{
get { return batchSize; }
set { batchSize = value; }
}
}
}
I also found that NHibernate is not doing batch inserts into PostgreSQL.
I identified two possible reasons:
1) Npgsql driver does not support batch inserts/updates (see forum)
2) NHibernate does not have *BatchingBatcher(Factory) for PostgreSQL(Npgsql). I tried using Devart dotConnect driver with NHibernate (I wrote custom driver for NHibernate) but it still did not worked.
I suppose this driver should also implement IEmbeddedBatcherFactoryProvider interface, but it seems not trivial for me (using one for Oracle did not worked ;) )
Did anybody managed to force Nhibarnate to do batch inserts to PostgreSQL or can confirm my conclusion?