sequelize: how to do equivalent of findAndCountAll for associations - orm

One of my biggest pains with sequelize is to basically "paginate and count" for associations.
Using findAndCountAll is fairly straightforward for one-off models, a bit painful for has-many, but utterly impossible to work with with many-to-many.
As an example, I have this many-to-many association established, where user can belong to many groups:
const UserGroup = db.define('user_groups', {
})
User.belongsToMany(Group, {through: UserGroup})
Group.belongsToMany(User, {through: UserGroup})
getting all users from a group would be straightforward:
user.getGroups().then....
but porting this to findAndCountAll just doesn't seem to work the same way:
User.findAndCountAll({
include: [{model: Group,
through: UserGroup,
where: { groupId : group.id,
userId : {$ne: user.id}}
}],
limit: limit,
offset: offset, ...
this doesn't work, as it associates keys from the where clause to the Group model.
I also tried:
User.findAndCountAll({
include: [{model: Group,
include: {
model: UserGroup,
where: { groupId : group.id,
userId : {$ne: user.id}}}
}],
limit: limit,
offset: offset, ...
but it fails as well, with Error: user_groups is not associated to groups!, which is not really true.
Is there a clean way to do this, preferably with the helper methods?

I had this problem earlier, and I decided to change functionality in this way...
const include = [
{
model: models.UserGroup,
attributes: [],
as: 'groups'
}
];
const [total, data] = await Promise.all([
User.count({ distinct: true, include }),
User.findAll({
order: [sequelize.literal(`MAX("groups"."createdAt") desc`)],
group: ['User.id'],
include
})
]);

I'm afraid there is no possibility to use findAndCountAll method when eager loading associations (or at least when you also want to add some conditions to associations). There is no option to get information about total count of associated items using, in your case, user.getGroups().
Although you can use Sequelize's shorthand method for counting association's instances, in your case, user.countGroups() before returning user.getGroups(). If you want to fetch related groups with some conditions, simply pass them as getGroups() method parameter (it takes same options parameter as standard findAll)

Related

Loopback join relation without selecting relation

I'am trying to build a query with loopback 4 with relation between 2 entities
customer.model.ts:
#model()
export class Customer extends Entity {
// id, name properties
#hasMany(() => Order)
orders?: Order[];
}
order.model.ts:
#model()
export class Order extends Entity {
// id, desc properties
#belongsTo(() => Customer)
customerId: Customer;
}
My goal is to get all cutomers that have at least 1 order but without selecting their orders, this my query
await customerRepository.find({
include: [{ relation: "orders" }],
});
I tried too :
await customerRepository.find({
include: [{ relation: "orders" }],
fields: {propertyName: }
});
Thank you for your help!
The behavior of excluding entries that doesn't have any related data (in your case excluding customers without any orders) is possible using INNER JOIN, which currently isn't supported in loopback by default.
You can use this newly published sequelize extension I recently worked on called
loopback4-sequelize which supports inner joins as well.
To achieve the expected results you'll need to set required to true as described here, so your filter object will look like this:
{
"include": [{
"relation": "orders",
"required": true, // returns only those customers who have orders
"scope": {
"fields": [] // exclude all fields in orders
}
}]
}
A point to keep in mind is that loopback by default never run SQL JOIN Queries internally. It uses inclusion resolvers which is responsible to include related models and is called after your source table's data is returned. Resulting in two different calls when you use include in your filter object.

Extracting additional data with query with keen.io

I have a (simplified) query that looks as follows.
var pageViews = new Keen.Query('count', {
eventCollection: 'Loaded a Page',
groupBy: 'company.id'
});
And use it as follows.
client.run(pageViews, function(result, error) {
// Do something here
});
This will give me the following JSON to work with:
{
"result": [
{
"company.id": 1,
"result": 3
},
{
"company.id": 2,
"result": 11
},
{
"company.id": 3,
"result": 7
}
]
}
However, I would also like to get back the name of each company, i.e. the company.name property. I looked through keen.io's documentation, and I could find no way of doing this. Is there a way to do this? Logically speaking, I don't see any reason why it would not be possible, but the question is if it has been implemented.
Grouping by multiple properties will get you what you're looking for:
var pageViews = new Keen.Query('count', {
eventCollection: 'Loaded a Page',
groupBy: ['company.id','company.name']
});
That being said, it's important to note that Keen is not an entity database. Keen is optimized to store and analyze event data, which is different than entity data. More complex uses of entity data may not perform well using this solution.

By code mapping of many-to-many with OrderBy

I'm using by code mappings and trying to map a manytomany. This works fine but I need OrderBy for the child collection items. I noticed this has been omitted (it does exist in the HBM mappings). e.g.
public class Mapping : EntityMapper<Category>
{
public Mapping()
{
Set(x => x.Items, m =>
{
m.Table("ItemCategories");
m.Key(k => k.Column("CategoryId"));
m.Inverse(true);
m.Cascade(Cascade.None);
}, col => col.ManyToMany(m =>
{
m.Columns(x => x.Name("ItemId"));
//m.OrderBy("Score desc"); // missing in Nh4.x?
}));
}
}
Is there a workaround for this? I tried following the suggestion in this article whereby I can set the property before the session factory is built but it has no effect. e.g.
cfg.GetCollectionMapping(typeof(Category).FullName + ".Items").ManyToManyOrdering = "Score desc";
cfg.BuildSessionFactory();
Am I doing something wrong or is OrderBy on manytomany not supported in Nh4?
Also, is it possible to restrict the maximum number of items retrieved in the collection?
Replaced the many to many with one to many and introduced an entity that represents the relationship (followed advice from this article).
This has the upside of allowing you to map the order-by column as well as other columns, and also solved the issue of restricting the number of items in the collection by using the one-to-many's Where() and Filter() clauses.

413 Error FULL HEAD Query URL too large Rally

I am using app SDK 2.0.I have the following code to retrieve the tasks for first 100 Team Members of the project for a particular Iteration
//getting the first 100 elements of the owners Array
owners = Ext.Array.splice(owners,0,100);
//Initial configuration of the filter object
var ownerFilter = Ext.create('Rally.data.QueryFilter', {
property: 'Owner.DisplayName',
operator:'=',
value: owners[0]
});
/*Creating the filter object to get all the tasks for 100 members in that project*/
Ext.Array.each(owners,function(member){
ownerFilter = ownerFilter.or(
Ext.create('Rally.data.QueryFilter',{
property: 'Owner.DisplayName',
operator:'=',
value: member
}));
});
//Iteration Filter for the Object
var iterationFilter = Ext.create('Rally.data.QueryFilter', {
property: 'Iteration.Name',
operator:'=',
value: 'Iteration 4.2'
});
var filter = ownerFilter.and(iterationFilter);
Rally.data.ModelFactory.getModel({
type: 'Task',
success: function(model) {
var taskStore = Ext.create('Rally.data.WsapiDataStore', {
model: model,
fetch:true,
limit:Infinity,
pageSize:200,
filters:[filter],
autoLoad:true,
projectScopeDown:true,
listeners:{
load:function(store,data,success) {
Ext.Array.each(data, function(record){
console.log(record.data);
});
}
}
});
}
});
This code is giving me a 413 error since the URL for the query is too large.The request URL has names of all 100 members of the project.How can I solve this problem ? Are there any efficient filtering options available?
Team Membership is a tough attribute to filter Artifacts on, since it resides as an attribute on Users and is a collection that can't be used in Queries.
In this case you might be better off doing more of the filtering client-side. Since Team Membership "approximates" or associates to Projects, you may wish to start by filtering your Tasks on Project, i.e. (Project.Name = "My Project"). Then narrow things down client side by looping through and ensuring each Task is owned by a member of the Team in your owners collection.
Alternatively, you might look to do something like use Tags on your Tasks that mirror the Team Names. You could then filter your Tasks server side by doing a query such as:
(Tags.Name contains "My Team")
I know these aren't ideal options and probably not the answer you're looking for, but, they are some ideas to avoid having to build a massive username-based query.

Sproutcore datasources and model relationships

I currently have a Sproutcore app setup with the following relationships on my models:
App.Client = SC.Record.extend({
name: SC.Record.attr(String),
brands: SC.Record.toMany('App.Brand', {isMaster: YES, inverse: 'client'})
});
App.Brand = SC.Record.extend({
name: SC.Record.attr(String),
client: SC.Record.toOne('App.Client, {isMaster: NO, inverse: 'brands'})
});
When I was working with fixtures my fixture for a client looked like this:
{
guid: 1,
name: 'My client',
brands: [1, 2]
}
And my fixture for a brand looked like this:
{
guid: 1,
name: 'My brand',
client: 1
}
Which all worked fine for me getting a clients brands and getting a brands client.
My question is in regards to how Datasources then fit into this and how the server response should be formatted.
Should the data returned from the server mirror exactly the format of the fixtures file? So clients should always contain a brands property containing an array of brand ids? And vice versa.
If I have a source list view which displays Clients with brands below them grouped. How would I go about loading that data for the source view with my datasource? Should I make a call to the server to get all the Clients and then follow that up with a call to fetch all the brands?
Thanks
Mark
The json you return will mostly mirror the fixtures. I recently had pretty much the same question as you, so I built a backend in Grails and a front end in SC, just to explore the store and datasources. My models are:
Scds.Project = SC.Record.extend(
/** #scope Scds.Project.prototype */ {
primaryKey: 'id',
name: SC.Record.attr(String),
tasks: SC.Record.toMany("Scds.Task", {
isMaster: YES,
inverse: 'project'
})
});
Scds.Task = SC.Record.extend(
/** #scope Scds.Task.prototype */ {
name: SC.Record.attr(String),
project: SC.Record.toOne("Scds.Project", {
isMaster: NO
})
});
The json returned for Projects is
[{"id":1,"name":"Project 1","tasks":[1,2,3,4,5]},{"id":2,"name":"Project 2","tasks":[6,7,8]}]
and the json returned for tasks, when I select a Project, is
{"id":1,"name":"task 1"}
obviously, this is the json for 1 task only. If you look in the projects json, you see that i put a "tasks" array with ids in it -- thats how the internals know which tasks to get. so to answer your first question, you dont need the id from child to parent, you need the parent to load with all the children, so the json does not match the fixtures exactly.
Now, it gets a bit tricky. When I load the app, I do a query to get all the Projects. The store calls the fetch method on the datasource. Here is my implementation.
Scds.PROJECTS_QUERY = SC.Query.local(Scds.Project);
var projects = Scds.store.find(Scds.PROJECTS_QUERY);
...
fetch: function(store, query) {
console.log('fetch called');
if (query === Scds.PROJECTS_QUERY) {
console.log('fetch projects');
SC.Request.getUrl('scds/project/list').json().
notify(this, '_projectsLoaded', store, query).
send();
} else if (query === Scds.TASKS_QUERY) {
console.log('tasks query');
}
return YES; // return YES if you handled the query
},
_projectsLoaded: function(response, store, query) {
console.log('projects loaded....');
if (SC.ok(response)) {
var recordType = query.get('recordType'),
records = response.get('body');
store.loadRecords(recordType, records);
store.dataSourceDidFetchQuery(query);
Scds.Statechart.sendEvent('projectsLoaded')
} else {
console.log('oops...error loading projects');
// Tell the store that your server returned an error
store.dataSourceDidErrorQuery(query, response);
}
}
This will get the Projects, but not the tasks. Sproutcore knows that as soon as I access the tasks array on a Project, it needs to get them. What it does is call retrieveRecords in the datasource. That method in turn calls retrieveRecord for every id in the tasks array. My retrieveRecord method looks like
retrieveRecord: function(store, storeKey) {
var id = Scds.store.idFor(storeKey);
console.log('retrieveRecord called with [storeKey, id] [%#, %#]'.fmt(storeKey, id));
SC.Request.getUrl('scds/task/get/%#'.fmt(id)).json().
notify(this, "_didRetrieveRecord", store, storeKey).
send();
return YES;
},
_didRetrieveRecord: function(response, store, storeKey) {
if (SC.ok(response)) {
console.log('succesfully loaded task %#'.fmt(response.get('body')));
var dataHash = response.get('body');
store.dataSourceDidComplete(storeKey, dataHash);
} ...
},
Note that you should use sc-gen to generate your datasource, because it provides a fairly well flushed out stub that guides you towards the methods you need to implement. It does not provide a retrieveMethods implementation, but you can provide your own if you don't want to do a single request for each child record you are loading.
Note that you always have options. If I wanted to, I could have created a Tasks query and loaded all the tasks data up front, that way I wouldn't need to go to my server when I clicked a project. So in answer to your second question, it depends. You can either load the brands when you click on the client, or you can load all the data up front, which is probably a good idea if there isn't that much data.