I had a scenario in Oracle where i need to match a substring part of column with a list of values. i was using sqlfunction projection for applying the substring on the required column, and added that projection as part of an In Clause Restriction. Below is the simplified criteria i wrote for that.
ICriteria criteriaQuery = session.CreateCriteria<Meeting>()
.Add(Restrictions.In(
Projections.SqlFunction(
"substring",
NHibernateUtil.String,
Projections.Property("Code"),
Projections.Constant(1),
Projections.Constant(3)),
new string[] { "D01", "D02" }))
.Add(Restrictions.In("TypeId", meetingTypes));
The problem that i had with this was that the generated SQL was wrong, where the number of parameters registered for the statement are more than what the statement actually uses and some parameters are repeated even though they are not used. This causes the statement to fail with the message - ORA-01036: illegal variable name/number.
Generated Query
SELECT this_.Meeting_id as Meeting1_0_2_, .....
WHERE substr(this_.Mcs_Main, :p0, :p1) in (:p2, :p3)
and this_.Meeting_Type_Id in (:p4, :p5);
:p0 = 1, :p1 = 3, :p2 = 1, :p3 = 3, :p4 = 'D02', :p5 = 'D03', :p6 = 101, :p7 = 102
p2 and p3 are generated again and are duplicates of p0, p1 because of which the entire query is failing.
I was able to temporarily resolve this by mapping a a new property with a formula, but i don't think that is the right approach since the formula will be executed always even when i don't need the substring to be evaluated.
Any suggestions on whether projections work fine when used with the combination of In clause, the same projection works fine when i use Equal Restriction and not In.
This bug is fixed in 3.0.0.GA version.
Related
In Dynamics 365 for Finance and Operations, they describe a method of creating SQL statements "as objects, as opposed to text", but this is somewhat of a lie. They use the objects to create the text which then populates str sqlStatement = selectExpr.getExpression(null);
This sqlStatement would then feed the obsolete statement.executeQuery(sqlStatement);.
I can make the warning go away by using executeQueryWithParameters() with an empty map (SqlParams::create()) as the second parameter, but this seems to be "cheating".
Is there a way I can/should refactor the following to populate the map correctly?
SQLBuilderSelectExpression selectExpression = SQLBuilderSelectExpression::construct();
selectExpression.parmUseJoin(true);
SQLBuilderTableEntry vendTable = selectExpression.addTableId(tableNum(VendTable));
SQLBuilderTableEntry dirPartyTable = vendTable.addJoinTableId(tableNum(DirPartyTable));
SQLBuilderFieldEntry accountNum = vendTable.addFieldId(fieldNum(VendTable, AccountNum));
SQLBuilderFieldEntry name = dirPartyTable.addFieldId(fieldNum(DirPartyTable, Name));
SQLBuilderFieldEntry dataAreaId = vendTable.addFieldId(fieldNum(VendTable, dataAreaId));
SQLBuilderFieldEntry blocked = vendTable.addFieldId(fieldNum(VendTable, Blocked));
vendTable.addRange(dataAreaId, curext());
vendTable.addRange(blocked, CustVendorBlocked::No);
selectExpression.addSelectFieldEntry(SQLBuilderSelectFieldEntry::newExpression(accountNum, 'AccountNum'));
selectExpression.addSelectFieldEntry(SQLBuilderSelectFieldEntry::newExpression(name, 'Name'));
str sqlStatement = selectExpression.getExpression(null);
// FIXME:
ResultSet resultSet = statement.executeQueryWithParameters(sqlStatement, SqlParams::create());
Below is how you would write your code as a standard X++ query. However, I must note that what you're doing may not be the best approach.
DirPartyTable is a special table in AX as it supports inheritance, so you should make sure you fully understand the framework. See:
https://learn.microsoft.com/en-us/dynamicsax-2012/appuser-itpro/implementing-the-global-address-book-framework-white-paper
https://learn.microsoft.com/en-us/dynamics365/fin-ops-core/fin-ops/organization-administration/overview-global-address-book
Code:
VendTable vendTable;
DirPartyTable dirPartyTable;
while select AccountNum from vendTable
where vendTable.Blocked == CustVendorBlocked::No
// DataAreaId along with Partition, are automatically included in the query context depending
// on the company context you're executing the code from
// && vendTable.dataAreaId == curext()
join Name from dirPartyTable
where dirPartyTable.RecId == vendTable.Party
{
info(strFmt("Account: %1; Name: %2", vendTable.AccountNum, dirPartyTable.Name));
}
Regarding an AOT query, look in the AOT at \Queries\VendTableListPage and expand the data sources and learn from it.
Regardless of what OP is trying to do with the query, the answer to the question of "how do I correctly replace executeQuery with executeQueryWithParameters" can be found in the following article.
https://learn.microsoft.com/en-us/dynamics365/fin-ops-core/dev-itpro/dev-ref/query-with-parameters
The new *WithParameters APIs were introduced as a way to mitigate sql injection attacks which may occur when building up sql strings manually with un-sanitized sql parameters as input.
Snippet of the code example from above doc shows how to correctly populate the map to match the sql statement:
str sql = #"
UPDATE Wages
SET Wages.Wage = Wages.Wage * #percent
WHERE Wages.Level = #Level";
Map paramMap = SqlParams::create();
paramMap.add('percent', 1.1); // 10 percent increase
paramMap.add('Level', 'Manager'); // Management increase
int cnt = statement.executeUpdateWithParameters(sql, paramMap);
How can I transform this sql to codeigniter query? I have screenshot of an EER diagram to help you understand better. So there are three tables to join. I guess, one is "pjesma", then "izvodjac", and "izvodi_pjesmu". This sql works when I run it and gives me results I want. I am also using pagination so I need limit and offset somehow included.
SELECT pjesma_id, naslov, naziv
FROM pjesma p, izvodjac i, izvodi_pjesmu ip
WHERE p.pjesma_id = ip.pjesma_pjesma_id
AND i.izvodjac_id = ip.izvodjac_izvodjac_id
in model:
public function paginacija_pjesme($limit, $offset) {
$this->db->select('pjesma_id', 'naslov', 'naziv');
$this->db->from('pjesma p');
$this->db->join('izvodi_pjesmu ip', 'p.pjesma_id=ip.pjesma_pjesma_id');
$this->db->join('izvodjac i', 'i.izvodjac_id=ip.izvodjac_izvodjac_id');
$this->db->limit($limit, $offset);
$query = $this->db->get();
return $query->result();
}
EDIT:
So in select I used wrong syntax, this line:
$this->db->select('pjesma_id', 'naslov', 'naziv');
should be like this:
$this->db->select('pjesma_id, naslov, naziv');
A few things, as you are dealing with objects within the query builder class, you can method-chain and make things a little less cluttered.
You are maybe getting an ambiguous column error, joining and not prefixing your select. Hard to tell without the error being posted you are facing.
Always best to (just in case) set defaults to your method arguments as well, in case you pass a blank variable (or don't need to for whatever reason).
When you are using multiple tables in the FROM statement you are actually doing a CROSS JOIN, and your new code is giving INNER JOIN,trying the join syntax direct to your SQL server might help as you are not actually doing the same thing between your first written query, and the codeigniter one..
Try the below with LEFT join just to see... If there are NULL joins using INNER they will be omitted
public function paginacija_pjesme($limit = 10, $offset = 0) {
$query = $this->db->select('p.pjesma_id, p.naslov, i.naziv')
->from('pjesma p')
->join('izvodi_pjesmu ip', 'p.pjesma_id = ip.pjesma_pjesma_id', 'left')
->join('izvodjac i', 'i.izvodjac_id = ip.izvodjac_izvodjac_id', 'left')
->limit($limit, $offset)
->get();
return $query->result();
}
Are you getting any errors? what do you see if you print_r($query->result()) ?
try also spitting out $this->db->last_query() - this will output the SQL codeigniter has build. Then running this directly to your database, and see the results from there. You might even see the issue without needing to run it.
This is an example of what your current codeigniter code is generating (as you can see, different to your original query):
SELECT pjesma_id, naslov, naziv
FROM pjesma p
INNER JOIN izvodi_pjesmu ip ON p.pjesma_id = ip.pjesma_pjesma_id
INNER JOIN izvodjac i ON i.izvodjac_id = ip.izvodjac_izvodjac_id
# LIMIT 10,0
I have following sql query in my hbm file. The SCHEMA, A and B are schema and two tables.
select
*
from SCHEMA.A os
inner join SCHEMA.B o
on o.ORGANIZATION_ID = os.ORGANIZATION_ID
where
case
when (:pass = 'N' and os.ORG_ID in (:orgIdList)) then 1
when (:pass = 'Y') then 1
end = 1
and (os.ORG_SYNONYM like :orgSynonym or :orgSynonym is null)
This is a pretty simple query. I had to use the case - when to handle the null value of "orgIdList" parameter(when null is passed to sql IN it gives error). Below is the relevant java code which sets the parameter.
if (_orgSynonym.getOrgIdList().isEmpty()) {
query.setString("orgIdList", "pass");
query.setString("pass", "Y");
} else {
query.setString("pass", "N");
query.setParameterList("orgIdList", _orgSynonym.getOrgIdList());
}
This works and give me the expected output. But I would like to know if there is a better way to handle this situation(orgIdList sometimes become null).
There must be at least one element in the comma separated list that defines the set of values for the IN expression.
In other words, regardless of Hibernate's ability to parse the query and to pass an IN(), regardless of the support of this syntax by particular databases (PosgreSQL doesn't according to the Jira issue), Best practice is use a dynamic query here if you want your code to be portable (and I usually prefer to use the Criteria API for dynamic queries).
If not need some other work around like what you have done.
or wrap the list from custom list et.
I am using Doctrine2 and Zf2 , now when I need to fetch count of rows, I have got the following two ways to fetch it. But my worry is which will be more optimized and faster way, as in future the rows would be more than 50k. Any suggestions or any other ways to fetch the count ?? Is there any function to get count which can be used with findBy ???
Or should I use normal Zf2 Database library to fetch count. I just found that ORM is not preferred to fetch results when data is huge. Please any help would be appreciated
$members = $this->getEntityManager()->getRepository('User\Entity\Members')->findBy(array('id' => $id, 'status' => '1'));
$membersCnt = sizeof($members);
or
$qb = $this->getEntityManager()->createQueryBuilder();
$qb->select('count(p)')
->from('User\Entity\Members', 'p')
->where('p.id = '.$id)
->andWhere('p.status = 1');
$membersCnt = $qb->getQuery()->getSingleScalarResult();
Comparison
1) Your EntityRepository::findBy() approach will do this:
Query the database for the rows matching your criteria. The database will return the complete rows.
The database result is then transformed (hydrated) into full PHP objects (entities).
2) Your EntityManager::createQueryBuilder() approach will do this:
Query the database for the number of rows matching your criteria. The database will return a simple number (actually a string representing a number).
The database result is then transformed from a string to a PHP integer.
You can safely conclude that option 2 is far more efficient than option 1:
The database can optimize the query for counting, which might make the query faster (take less time).
Far less data is returned from the database.
No entities are hydrated (only a simple string to integer cast).
All in all less processing power and less memory will be used.
Security comment
Never concatenate values into a query!
This can make you vulnerable to SQL injection attacks when those values are (derived from) user-input.
Also, Doctrine2 can't make use of prepared statements / parameter binding, which can lead to some performance-loss when the same query is used often (with or without different parameters).
In other words, replace this:
->where('p.id = '.$id)
->andWhere('p.status = 1')
with this:
->where('p.id = :id')
->andWhere('p.status = :status')
->setParameters(array('id' => $id, 'status' => 1))
or:
->where($qb->expr()->andX(
$qb->expr()->eq('p.id', ':id'),
$qb->expr()->eq('p.status', ':status')
)
->setParameters(array('id' => $id, 'status' => 1))
Additionally
For this particular query, there's no need to use the QueryBuilder, you can use straight DQL in stead:
$dql = 'SELECT COUNT(p) FROM User\Entity\Members p WHERE p.id = :id AND p.status = :status';
$q = $this->getEntityManager()->createQuery($dql);
$q->setParameters(array('id' => $id, 'status' => 1));
$membersCnt = $q->getSingleScalarResult();
You should totally go to the dql version of the count.
With the first method you will hydrate (convert from db resultset to objects) each of the rows as single object and put them on one array and then count the amount items in that array. That will be a totally waste of memory and cycles if the only objective is to know the number of elements in that result set.
With the second method the dql will be gracefully converted to SELECT COUNT(*) Blah blah blah
plain SQL sentence and will retrieve directly the count from db.
The comment about ORM is not preferred to when to retrieve data is huge is true, in big batch process you should paginate your query to retrieve data instead all at the same time to avoid memory overrides but in that case you are only retrieving a single number, the total count so this rule doesn’t apply.
Query builder is so slow .
Use DQL for faster select .
$query = $this->getEntityManager()->createQuery("SELECT count(m) FROM User\Entity\Members m WHERE m.status = 1 AND m.id = :id ");
$query->setParameter(':id', $id);
You need setParameter for prevent SQL injection .
Stored procedure is fastest but it depend on your DB .
Make all relations of entity Lazy.
say i have a nvarchar field in my database that looks like this
1, "abc abccc dabc"
2, "abccc dabc"
3, "abccc abc dabc"
i need a select LINQ query that would match the word "abc" with boundaries not part of a string
in this case only row 1 and 3 would match
from row in table.AsEnumerable()
where row.Foo.Split(new char[] {' ', '\t'}, StringSplitOptions.None)
.Contains("abc")
select row
It's important to include the call to AsEnumerable, which means the query is executed on the client-side, else (I'm pretty sure) the Where clause won't get converted into SQL succesfully.
Maybe a regular expression like this (nb - not compiled or tested):
var matches = from a in yourCollection
where Regex.Match(a.field, ".*\sabc\s.*")
select a;
datacontext.Table.Where(
e => Regex.Match(e.field, #"(.*?[\s\t]|^)abc([\s\t].*?|$)")
);
or
datacontext.Table.Where(
e => e.Split(' ', '\t').Contains("abc");
);
For efficiency, you want to do as much of the filtering as possible on the server, and then the rest of the filtering on the client. You can't use Regex on the server (SQL Server doesn't support it) so the solution is to first use a LIKE-type search (by calling .Contains) then use Regex on the client to further refine the results:
db.MyTable
.Where (t => t.MyField.Contains ("abc"))
.AsEnumerable() // Executes locally from this point on
.Where (t => Regex.IsMatch (t.MyField, #"\babc\b"))
This ensures that you retrieve only the rows from SQL Server than contain the letters 'abc' (regardless of whether they're a word-boundary match or not) and use Regex on the client-side to further restrict the result set so that only matches that are on word boundaries are included.