I am trying to working on a query builder and the example/content is just imaginary.
query = from en in Data.InfectionRevision
query = from q in query, select: map(q, [:is_complete, :is_active])
query = from(q in query,select_merge: %{"$aggregate": %{"$max": %{^"revision_id" => max(q.revision_id)}}})
query = from(q in query,group_by: q.encounter_id,
select_merge: %{"$group" => %{^"encounter_id" => q.encounter_id}})
But the query I get is having incorrect key for group, although I have passed correct key while building query. this is what I got
#Ecto.Query<from i0 in Data.InfectionRevision, group_by: [i0.encounter_id],
select: merge(merge(
map(i0, [:is_complete, :is_active]),
%{"$aggregate": %{"$max": %{^"revision_id" => max(i0.revision_id)}}}),
%{"$group" => %{^"revision_id" => i0.encounter_id}})>
%{"$group" => %{^"revision_id" => i0.encounter_id}} should be actually %{"$group" => %{^"encounter_id" => i0.encounter_id}}
What Step or part of the query I am building wrong? How to fix it?
Note:
sample project on which you can run/build above queries
Please ignore the query content, it might not makes sense because its just an example and should work dynamically as its part of dynamic query builder
I need to build query step by step from passed params dynamically
Related
I am working in yii2.
There are employee and company table employee contains company_id.
I have a filter search running properly If I use joinWith()
$query = Employee::find();
$query->joinWith(['company']);
$dataProvider = new ActiveDataProvider([
'query' => $query,
'pagination' => false,
'sort' => false,
]);
if (!($this->load($params) && $this->validate())) {
return $dataProvider;
}
//and below is the filterwhere
$query->andFilterWhere(['like', 'company.name', $this->company_id]);
But issue came when I make a query using with()
$query = Employee::find()->with(['company']);
$dataProvider = new ActiveDataProvider([
'query' => $query,
'pagination' => false,
'sort' => false,
]);
if (!($this->load($params) && $this->validate())) {
return $dataProvider;
}
//when query contain with() then this filter is not working.
$query->andFilterWhere(['like', 'company.name', $this->company_id]);
This gives error when I use with()
Database Exception – yii\db\Exception
SQLSTATE[42S22]: Column not found: 1054 Unknown column 'company.name' in 'where clause'
The SQL being executed was: SELECT COUNT(*) FROM `employee` WHERE `company`.`name` LIKE '%1%'
Here is the relation in employee with company:
public function getCompany(){
return $this->hasOne(Company::className(), ['id'=> 'company_id']);
}
Can anyone help me or guide me how could I filter data properly using with() in a query?
Thanks.
You can't swap joinWith() and with() methods when you need to filter by the column from the related table. That's because these methods does completely different things.
Methods like joinWith() and join() actually modifies the query to add the "JOIN" part to the SQL query. The with in joinWith allows you to specify the joined table by the relation definition in the model. The eager loading in joinWith is only side effect and you can even turn that off by passing false as second parameter.
When you do:
Employee::find()->joinWith(['company'])->all();
The query that is run looks like:
SELECT * FROM employee LEFT JOIN company ON (...)
On the other side the method with() doesn't modify the query itself. It only forces the eager loading of related models. In reality the second query is used for preloading the related records.
When you do:
Employee::find()->with(['company'])->all();
It actually runs queries like these:
SELECT * FROM employee;
SELECT * FROM company WHERE id IN (...company ids selected in first query...);
So when you try to do:
$query = Employee::find()
->with(['company'])
->andFilterWhere(['like', 'company.name', $this->company_id])
->all();
The generated query is
SELECT * FROM employee WHERE company.name LIKE ...
A search term comes from UI to search entities of a table. The order that these search results should show up in UI is like this:
First: exact match
Second: starts with that term
Third: contains a word of that term
Forth: ends with that term
Fifth: contains the term in any matter
So I first got the entities from DB:
result = entities.Where(e => e.Name.Contains(searchTerm)).ToList();
And then I rearranged them in memory:
var sortedEntities = result.Where(e => e.Name.ToLower() == searchTerm.ToLower())
.Union(result.Where(e => e.Name.StartsWith(searchTerm, StringComparison.OrdinalIgnoreCase)))
.Union(result.Where(e => e.Name.Contains($" {searchTerm} ")))
.Union(result.Where(e => e.Name.EndsWith(searchTerm, StringComparison.OrdinalIgnoreCase)))
.Union(result.Where(e => e.Name.Contains(searchTerm)));
It was working fine until I added paging. Now if an exact match is on page 2 (in data coming from DB) it won't show up first.
The only solution I can think of is to separate the requests (so 5 requests in this case) and keep track of page size manually. My question is that is there a way to tell DB to respect that order and get the sorted data in one DB trip?
It took me some time to realize that you use Union in an attempt to order data by "match strength": first the ones that match exactly, then the ones that match with different case, etc. When I see Unions with predicates my Pavlov-conditioned mind translates it into ORs. I had to switch from thinking fast to slow.
So the problem is that there is no predictable sorting. No doubt, the chained Union statements do produce a deterministic final sort order, but it's not necessarily the order of the Unions, because each Union also executes an implicit Distinct. The general rule is, if you want a specific sort order, use OrderBy methods.
Having said that, and taking...
var result = entities
.Where(e => e.Name.Contains(searchTerm))
.Skip((pageNumber - 1) * pageSize)
.Take(pageSize).ToList();
...the desired result seems to be obtainable by:
var sortedEntities = result
.OrderByDescending(e => e.Name == searchTerm)
.ThenByDescending(e => e.Name.ToLower() == searchTerm.ToLower())
.ThenByDescending(e => e.Name.StartsWith(searchTerm, StringComparison.OrdinalIgnoreCase))
... etc.
(descending, because false orders before true)
However, if there are more matches than pageSize the ordering will be too late. If pageSize = 20 and item 21 is the first exact match this item will not be on page 1. Which means: the ordering should be done before paging.
The first step would be to remove the .ToList() from the first statement. If you remove it, the first statement is an IQueryable expression and Entity Framework is able to combine the full statement into one SQL statement. The next step would be to move Skip/Take to the end of the full statement and it'll also be part of the SQL.
var result = entities.Where(e => e.Name.Contains(searchTerm));
var sortedEntities = result
.OrderByDescending(e => e.Name == searchTerm)
.ThenByDescending(e => e.Name.ToLower() == searchTerm.ToLower())
.ThenByDescending(e => e.Name.StartsWith(searchTerm, StringComparison.OrdinalIgnoreCase))
... etc
.Skip((pageNumber - 1) * pageSize)
.Take(pageSize).ToList();
But now a new problem blows in.
Since string comparison with StringComparison.OrdinalIgnoreCase isn't supported Entity Framework will auto-switch to client-side evaluation for part of the statement. All of the filtered results will be returned from the database, but most of the the ordering and all of the paging will be done in memory.
That may not be too bad when the filter is narrow, but very bad when it's wide. So, ultimately, to do this right, you have to remove StringComparison.OrdinalIgnoreCase and settle with a somewhat less refined match strength. Bringing us to the
End result:
var result = entities.Where(e => e.Name.Contains(searchTerm));
var sortedEntities = result
.OrderByDescending(e => e.Name == searchTerm)
.ThenByDescending(e => e.Name.StartsWith(searchTerm))
.ThenByDescending(e => e.Name.Contains($" {searchTerm} "))
.ThenByDescending(e => e.Name.EndsWith(searchTerm))
.ThenByDescending(e => e.Name.Contains(searchTerm))
.Skip((pageNumber - 1) * pageSize)
.Take(pageSize).ToList();
Why "less refined"? Because, according your comments, the database collation isn't case sensitive, so SQL can't distinguish exact matches by case without adding COLLATE statements. That's something we can't do with LINQ.
I'm using CakePHP 3.5.13 to build an application which has 4 separate databases.
The main database (Datasource default in config/app.php) for the application has been baked. It was a legacy database, and the naming conventions are not written according to the way CakePHP specifies. Nonetheless it works, after going through the Models and editing things.
In a controller I have the following:
$substances = TableRegistry::get('Substances');
$query = $substances->find()->limit(250)->offset(0);
$query->select(['id', 'app_id', 'name']);
$query->contain([
'Cas' => [
'sort' => ['Cas.id' => 'ASC']
]
]);
$query->contain([
'Ecs' => [
'sort' => ['Ecs.id' => 'ASC']
]
]);
If I var_dump($query) I get an SQL string as follows:
SELECT Substances.id AS `Substances__id`,
Substances.app_id AS `Substances__app_id`,
Substances.name AS `Substances__name`
FROM substances Substances LIMIT 250 OFFSET 0
I need to modify this so the query contains an INNER JOIN to a table which is stored in one of the other databases (Datasource sdb5_tmpdata in config/app.php). The SQL I need is as follows:
SELECT Substances.id AS `Substances__id`,
Substances.app_id AS `Substances__app_id`,
Substances.name AS `Substances__name`,
Substances.date AS `Substances__date`
FROM substances Substances
INNER JOIN `sdb5_tmpdata`.`searching_1745` AS tf
ON tf.id = Substances.id LIMIT 250 OFFSET 0;
The difference between the above query and original is the following SQL:
INNER JOIN `sdb5_tmpdata`.`searching_1745` AS tf
ON tf.id = Substances.id
I don't have a corresponding Model for the table 'searching_1745' because these tables are dynamically created (and later dropped) on a database which holds "temporary" user data.
Is it possible to modify the query object, $query, such that I can introduce the custom SQL that does the inner join?
I have tried $query = $query->newExpr()->add('INNER JOIN 'sdb5_tmpdata'.'searching_1745' AS tf ON tf.id = Substances.id'); but it doesn't work.
the query builder let you build the query as you like it. It does not matter if you don't have a Table Object for that table
$query->join([
'tf ' => [
'table' => 'sdb5_tmpdata.searching_1745',
'type' => 'INNER',
'conditions' => 'tf.id = Substances.id',
]);
see here
I don't know if you can modify the query object, but tou can always write your own custom queries:
use Cake\Datasource\ConnectionManager;
$conn = ConnectionManager::get('default');
$query = $conn->query('query goes here');
for more info, read: https://book.cakephp.org/3.0/en/orm/database-basics.html#executing-queries
I need to conditionally add a filter to particular dates in a query. There are common preconditions and the filter will be the same. Therefore I would like the common code to be in a method which can perform these checks and then have the consumer pass in the property which the filter should be applied to (could be applied to multiple).
Here is a simplified version of my code.
var query = dbContext.Documents.AsQueryable();
query = FilterDocumentsByDate(query, x => x.CreatedDate);
query = FilterDocumentsByDate(query, x => x.SubmittedDate);
private IQueryable<Document> FilterDocumentsByDate(IQueryable<Document> query, Func<Document, DateTime> propertyToSearch)
{
query = query.Where(x => propertyToSearch(x).Year > 2000);
return query;
}
When I look at the query in SQL profiler, I can see that the query is missing the WHERE clause (so all documents are being retrieved and the filter is being done in memory). If I copy/paste the code inline for both dates (instead of calling the method twice) then the WHERE clause for the both dates are included in the query.
Is there no way to add a WHERE condition to an IQueryable by passing a property in a Func which can be properly translated to SQL by Entity Framework?
EF is unable to understand your query, so it breaks and executes WHERE clause in memory.
The solution is creating dynamic expressions.
var query = dbContext.Documents.AsQueryable();
query = FilterDocumentsByDate(query, x => x.CreatedDate.Year);
query = FilterDocumentsByDate(query, x => x.SubmittedDate.Year);
private IQueryable<Document> FilterDocumentsByDate(IQueryable<Document> query, Expression<Func<Document, int>> expression)
{
var parameter = expression.Parameters.FirstOrDefault();
Expression comparisonExpression = Expression.Equal(expression.Body, Expression.Constant(2000));
Expression<Func<Document, bool>> exp = Expression.Lambda<Func<Document, bool>>(comparisonExpression, parameter);
query = query.Where(exp);
return query;
}
I am sorry, I haven't run this myself, but this should create WHERE statement. Let me know how it goes.
I am trying to join table prospects with leads. I am executing this query
$queryProspects = new SugarQuery();
$queryProspects->from(BeanFactory::getBean('Prospects'));
$leads = $queryProspects->joinTable('leads');
$queryProspects->select("prospects.id","prospects.lead_id");
$queryProspects->where()->equals("lead_id","117c3d5d-07d9-0ae7-5610-573ac87c9a35");
Before executing it like this.
$queryProspects->execute();
I am compiling my query like
$queryProspects->compileSql();
This query is not working after executing. query result after compiling is
SELECT prospects.id id, prospects.lead_id lead_id FROM prospects JOIN leads ON () WHERE prospects.deleted = 0 AND prospects.lead_id = '117c3d5d-07d9-0ae7-5610-573ac87c9a35'
I know the error is () WHERE which I need to remove, but unable to do changes in sugar query in order to remove these brackets and where clause (which are showing in sql generated query).
Change from and join statement like this.
$queryProspects->from(BeanFactory::getBean('Prospects'), array('team_security' => false));
$leads = $queryProspects->join('lead')->joinName();
lead in join is the link (name field) in your prospects > vardefs.php as shown below.
'lead' => array(
'name' => 'lead',
'type' => 'link',
'relationship' => 'lead_prospect',
'module' => 'Leads',
'source' => 'non-db',
'vname' => 'LBL_LEAD',
),
please execute your query like this :
$result = $queryProspects->execute();
For more details Follow this link :
Sugar Query