prisma Order by relation has only _count property. Can not order by relation fields - sql

consider following Prisma schema:
model Conversation {
id Int #id #default(autoincrement())
createdAt DateTime #db.Timestamp(6)
messages ConversationMessage[]
}
model ConversationMessage {
id Int #id #default(autoincrement())
text String #db.VarChar(1000)
sentAt DateTime #map("sent_at") #db.Timestamp(6)
conversationId Int? #map("conversation_id")
userId Int? #map("user_id")
conversation Conversation? #relation(fields: [conversationId], references: [id])
sender User? #relation(fields: [userId], references: [id])
}
I want to run such query so that I get a list of conversations ordered by date of their messages, i.e. the ones with new messages first.
prisma.conversation.findMany({
orderBy: {
messages: {
sentAt: 'desc'
}
},
...
})
But the only way that I can query now is like this, i.e. relation has only _count property somehow.
prisma.conversation.findMany({
orderBy: {
messages: {
'_count': 'desc'
}
},
...
})
Environment & setup
OS: Mac OS,
Database: PostgreSQL
Node.js version: v12.19.0
Prisma Version
prisma : 2.24.1
#prisma/client : 2.24.1
Current platform : darwin
Query Engine : query-engine 18095475d5ee64536e2f93995e48ad800737a9e4 (at node_modules/#prisma/engines/query-engine-darwin)
Migration Engine : migration-engine-cli 18095475d5ee64536e2f93995e48ad800737a9e4 (at node_modules/#prisma/engines/migration-engine-darwin)
Introspection Engine : introspection-core 18095475d5ee64536e2f93995e48ad800737a9e4 (at node_modules/#prisma/engines/introspection-engine-darwin)
Format Binary : prisma-fmt 18095475d5ee64536e2f93995e48ad800737a9e4 (at node_modules/#prisma/engines/prisma-fmt-darwin)
Default Engines Hash : 18095475d5ee64536e2f93995e48ad800737a9e4
Studio : 0.397.0
Preview Features : orderByRelation
Thank You!

While Prisma V2.19 introduced sort by relation aggregate value, as of this writing, the only aggregate property supported is count. To the best of my knowledge, what you are asking for is not directly supported by Prisma at the moment. It would be possible if they add min and max aggregate properties for sorting.
A possible workaround is to sort the messages inside Node.js after retrieval. I'm adding a solution that uses the orderByRelation preview feature to simplify the sorting and ensure the messages in a conversation are always ordered (newest first).
Updating Prisma Client to use orderByRelation preview feature.
First, update schema.prisma to add the preview feature
generator client {
provider = "prisma-client-js"
previewFeatures = ["orderByRelation"]
}
Now update the prisma client
prisma generate client
Get conversations and sort them by most recent message
// Assuming inside an async function
let unsortedConversations = await prisma.conversation.findMany({
include: {
messages: {
orderBy: {
sentAt: 'desc' // messages for each converastion will be ordered newest first.
}
}
},
// other conditions
})
unsortedConversations contains all required conversations, but they are unordered. You can sort it in the desired order by creating a custom comparator function.
function conversationComparatorFunction(conversationA, conversationB) {
// Conversations with 0 messages will be placed last in arbitrary order.
if (!conversationB.messages.length) return 1;
if (!conversationA.messages.length) return -1;
// sort conversations based on sentAt date of the first message.
// since messages were previously sorted, messages[0] always contain the most recent message.
if (conversationA.messages[0].sentAt > conversationB.messages[0].sentAt) {
return -1;
} else if (conversationA.messages[0].sentAt < conversationB.messages[0].sentAt) {
return 1;
} else return 0;
}
let sortedConversations = unsortedConversations.sort(conversationComparatorFunction)
Be warned though, if the number of Conversation records is very large sorting on the application side could lead to poor performance, especially considering Node.js is single-threaded.

OrderBy relation is still a preview feature - you need to make sure to use the feature flag
generator client {
provider = "prisma-client-js"
previewFeatures = ["orderByRelation"]
}
https://www.prisma.io/docs/concepts/components/prisma-client/filtering-and-sorting#sort-by-relation-preview

Related

How do perform a graph query and join?

I apologize for the title, I don't exactly know how to word it. But essentially, this is a graph-type query but I know RavenDB's graph functionality will be going away so this probably needs to be solved with Javascript.
Here is the scenario:
I have a bunch of documents of different types, call them A, B, C, D. Each of these particular types of documents have some common properties. The one that I'm interested in right now is "Owner". The owner field is an ID which points to one of two other document types; it can be a Group or a User.
The Group document has a 'Members' field which contains an ID which either points to a User or another Group. Something like this
It's worth noting that the documents in play have custom IDs that begin with their entity type. For example Users and Groups begin with user: and group: respectively. Example IDs look like this: user:john#castleblack.com or group:the-nights-watch. This comes into play later.
What I want to be able to do is the following type of query:
"Given that I have either a group id or a user id, return all documents of type a, b, or c where the group/user id is equal to or is a descendant of the document's owner."
In other words, I need to be able to return all documents that are owned by a particular user or group either explicitly or implicitly through a hierarchy.
I've considered solving this a couple different ways with no luck. Here are the two approaches I've tried:
Using a function within a query
With Dejan's help in an email thread, I was able to devise a function that would walk it's way down the ownership graph. What this attempted to do was build a flat array of IDs which represented explicit and implicit owners (i.e. root + descendants):
declare function hierarchy(doc, owners){
owners = owners || [];
while(doc != null) {
let ownerId = id(doc)
if(ownerId.startsWith('user:')) {
owners.push(ownerId);
} else if(ownerId.startsWith('group:')) {
owners.push(ownerId);
doc.Members.forEach(m => {
let owner = load(m, 'Users') || load(m, 'Groups');
owners = hierarchy(owner, owners);
});
}
}
return owners;
}
I had two issues with this. 1. I don't actually know how to use this in a query lol. I tried to use it as part of the where clause but apparently that's not allowed:
from #all_docs as d
where hierarchy(d) = 'group:my-group-d'
// error: method hierarchy not allowed
Or if I tried anything in the select statement, I got an error that I have exceeded the number of allowed statements.
As a custom index
I tried the same idea through a custom index. Essentially, I tried to create an index that would produce an array of IDs using roughly the same function above, so that I could just query where my id was in that array
map('#all_docs', function(doc) {
function hierarchy(n, graph) {
while(n != null) {
let ownerId = id(n);
if(ownerId.startsWith('user:')) {
graph.push(ownerId);
return graph;
} else if(ownerId.startsWith('group:')){
graph.push(ownerId);
n.Members.forEach(g => {
let owner = load(g, 'Groups') || load(g, 'Users');
hierarchy(owner, graph);
});
return graph;
}
}
}
function distinct(value, index, self){ return self.indexOf(value) === index; }
let ownerGraph = []
if(doc.Owner) {
let owner = load(doc.Owner, 'Groups') || load(doc.Owner, 'Users');
ownerGraph = hierarchy(owner, ownerGraph).filter(distinct);
}
return { Owners: ownerGraph };
})
// error: recursion is not allowed by the javascript host
The problem with this is that I'm getting an error that recursion is not allowed.
So I'm stumped now. Am I going about this wrong? I feel like this could be a subquery of sorts or a filter by function, but I'm not sure how to do that either. Am I going to have to do this in two separate queries (i.e. two round-trips), one to get the IDs and the other to get the docs?
Update 1
I've revised my attempt at the index to the following and I'm not getting the recursion error anymore, but assuming my queries are correct, it's not returning anything
// Entity/ByOwnerGraph
map('#all_docs', function(doc) {
function walkGraph(ownerId) {
let owners = []
let idsToProcess = [ownerId]
while(idsToProcess.length > 0) {
let current = idsToProcess.shift();
if(current.startsWith('user:')){
owners.push(current);
} else if(current.startsWith('group:')) {
owners.push(current);
let group = load(current, 'Groups')
if(!group) { continue; }
idsToProcess.concat(group.Members)
}
}
return owners;
}
let owners = [];
if(doc.Owner) {
owners.concat(walkGraph(doc.Owner))
}
return { Owners: owners };
})
// query (no results)
from index Entity/ByOwnerGraph as x
where x.Owners = "group:my-group-id"
// alternate query (no results)
from index Entity/ByOwnerGraph as x
where x.Owners ALL IN ("group:my-group-id")
I still can't use this approach in a query either as I get the same error that there are too many statements.

AND condition in where clause

I am trying to query the invoices with xero-ruby gem. I need to query with reference.
I build the query options as
opts = { statuses: %w[PAID DRAFT SUBMITTED AUTHORISED], where: { reference: ["Contains", job.quote_ref] } }
But on production it returns the following error:
Where clause invocation error: Operations on optional fields must be preceded by a null guard, e.g. \"[FieldName] != null AND [FieldName]...\""
I tried to add the null condition like this:
opts = { statuses: %w[PAID DRAFT SUBMITTED AUTHORISED], where: { reference: ["!=", nil] && ["Contains", job.quote_ref] } }
But since it is a hash, && condition doesn't work and it sends only the contains part.
I need help as Xero support does't help with ruby related issues
Tried building the query with multiple options:
opts = { statuses: %w[PAID DRAFT SUBMITTED AUTHORISED], where: { reference: ["!=", nil, "AND","contains", "1814"] }}
Not able to find a solution for this

Transcoding HTTP to gRPC: Same endpoint with different parameters

I already have a working gRPC project working. I'm looking to build an API to be able to do some HTTP requests.
I have the following 2 types:
message FindRequest {
ModelType model_type = 1;
oneof by {
string id = 2;
string name = 3;
}
}
message GetAllRequest {
ModelType model_type = 1;
int32 page_size = 2;
oneof paging {
int32 page = 3;
bool skip_paging = 4;
}
}
And then, I would like to have those 2 endpoints:
// Get a data set by ID or name. Returns an empty data set if there is no such
// data set in the database. If there are multiple data sets with the same
// name in the database it returns the first of these data sets.
rpc Find(FindRequest) returns (DataSet){
option (google.api.http) = { get: "/datasets" };
}
// Get (a page of) all data sets of a given type. If no page size is given
// (page <= 0) it defaults to 100. An unset page (page <= 0) defaults to the
// first page.
rpc GetAll(GetAllRequest) returns (GetAllResponse){
option (google.api.http) = { get: "/datasets" };
}
It makes sense to me to have 2 different endpoints with the same name, but that differ with the parameters.
For instance, requesting /datasets?model-type=XXX should be mapped to the GetAll function, while requesting /datasets?model-type=XXX&name=YYY should be mapped to Find function.
However, it doesn't work, since the mapping fails I guess, so none of these endpoints returns me anything.
I think the solution to make the mapping working would be to force the parameter to be required, however, I am working with proto3, which has disallowed the required field.
So how could I be able to have 2 endpoints with the same name, but different parameters, with proto3 ?
I know that if I am using different endpoint names, it is working, for example for the findRequest, I could have the following endpoint : /findDatasets, but regarding the best practice about API naming convention, it is not advisable, or is it ?
The conventional way to solve this problem is to use different methods. My hunch is that it's an anti-pattern to try to differentiate using the fields in the request string.
service YourService {
rpc FindSomething(FindSomethingRequest) returns (FindSomethingResponse){
option (google.api.http) = { get: "/something/find" };
}
rpc ListSomething(ListSomethingRequest) returns (ListSomethingResponse){
option (google.api.http) = { get: "/something/list" };
}
}
message FindSomethingRequest {
ModelType model_type = 1;
string id = 2;
string name = 3;
}
message ListSomethingRequest {
int32 page_size = 2;
int32 page_token = 3;
}
message ListSomethingResponse {
repeated ModelType model_types = 1;
int32 page_size = 2;
int32 next_page_token = 3;
}
I'm unsure of your underlying thing structure but, I think it's better practice to model things with all possible properties and permit leaving some unset (e.g. either id or name or possibly both in FindSomethingRequest) rather than creating different message types for all possible queries. You model the thing not how you interact with it.
In your implementation (!) of FindSomething, you then deal with the permutations of how users of the message may construct the fields. Perhaps reporting an error "Either id or name is required`.
I think ListSomething's messages could be simpler too. You request a List of (ModelTypes) and give a page_size and an page_token (that could be ""). It returns a list of ModelTypes, the size of the page returned (possibly less than requested) and a next_page_token if there is more data, that you can use to make the next ListSomething request.

Proper Way to Retrieve More than 128 Documents with RavenDB

I know variants of this question have been asked before (even by me), but I still don't understand a thing or two about this...
It was my understanding that one could retrieve more documents than the 128 default setting by doing this:
session.Advanced.MaxNumberOfRequestsPerSession = int.MaxValue;
And I've learned that a WHERE clause should be an ExpressionTree instead of a Func, so that it's treated as Queryable instead of Enumerable. So I thought this should work:
public static List<T> GetObjectList<T>(Expression<Func<T, bool>> whereClause)
{
using (IDocumentSession session = GetRavenSession())
{
return session.Query<T>().Where(whereClause).ToList();
}
}
However, that only returns 128 documents. Why?
Note, here is the code that calls the above method:
RavenDataAccessComponent.GetObjectList<Ccm>(x => x.TimeStamp > lastReadTime);
If I add Take(n), then I can get as many documents as I like. For example, this returns 200 documents:
return session.Query<T>().Where(whereClause).Take(200).ToList();
Based on all of this, it would seem that the appropriate way to retrieve thousands of documents is to set MaxNumberOfRequestsPerSession and use Take() in the query. Is that right? If not, how should it be done?
For my app, I need to retrieve thousands of documents (that have very little data in them). We keep these documents in memory and used as the data source for charts.
** EDIT **
I tried using int.MaxValue in my Take():
return session.Query<T>().Where(whereClause).Take(int.MaxValue).ToList();
And that returns 1024. Argh. How do I get more than 1024?
** EDIT 2 - Sample document showing data **
{
"Header_ID": 3525880,
"Sub_ID": "120403261139",
"TimeStamp": "2012-04-05T15:14:13.9870000",
"Equipment_ID": "PBG11A-CCM",
"AverageAbsorber1": "284.451",
"AverageAbsorber2": "108.442",
"AverageAbsorber3": "886.523",
"AverageAbsorber4": "176.773"
}
It is worth noting that since version 2.5, RavenDB has an "unbounded results API" to allow streaming. The example from the docs shows how to use this:
var query = session.Query<User>("Users/ByActive").Where(x => x.Active);
using (var enumerator = session.Advanced.Stream(query))
{
while (enumerator.MoveNext())
{
User activeUser = enumerator.Current.Document;
}
}
There is support for standard RavenDB queries, Lucence queries and there is also async support.
The documentation can be found here. Ayende's introductory blog article can be found here.
The Take(n) function will only give you up to 1024 by default. However, you can change this default in Raven.Server.exe.config:
<add key="Raven/MaxPageSize" value="5000"/>
For more info, see: http://ravendb.net/docs/intro/safe-by-default
The Take(n) function will only give you up to 1024 by default. However, you can use it in pair with Skip(n) to get all
var points = new List<T>();
var nextGroupOfPoints = new List<T>();
const int ElementTakeCount = 1024;
int i = 0;
int skipResults = 0;
do
{
nextGroupOfPoints = session.Query<T>().Statistics(out stats).Where(whereClause).Skip(i * ElementTakeCount + skipResults).Take(ElementTakeCount).ToList();
i++;
skipResults += stats.SkippedResults;
points = points.Concat(nextGroupOfPoints).ToList();
}
while (nextGroupOfPoints.Count == ElementTakeCount);
return points;
RavenDB Paging
Number of request per session is a separate concept then number of documents retrieved per call. Sessions are short lived and are expected to have few calls issued over them.
If you are getting more then 10 of anything from the store (even less then default 128) for human consumption then something is wrong or your problem is requiring different thinking then truck load of documents coming from the data store.
RavenDB indexing is quite sophisticated. Good article about indexing here and facets here.
If you have need to perform data aggregation, create map/reduce index which results in aggregated data e.g.:
Index:
from post in docs.Posts
select new { post.Author, Count = 1 }
from result in results
group result by result.Author into g
select new
{
Author = g.Key,
Count = g.Sum(x=>x.Count)
}
Query:
session.Query<AuthorPostStats>("Posts/ByUser/Count")(x=>x.Author)();
You can also use a predefined index with the Stream method. You may use a Where clause on indexed fields.
var query = session.Query<User, MyUserIndex>();
var query = session.Query<User, MyUserIndex>().Where(x => !x.IsDeleted);
using (var enumerator = session.Advanced.Stream<User>(query))
{
while (enumerator.MoveNext())
{
var user = enumerator.Current.Document;
// do something
}
}
Example index:
public class MyUserIndex: AbstractIndexCreationTask<User>
{
public MyUserIndex()
{
this.Map = users =>
from u in users
select new
{
u.IsDeleted,
u.Username,
};
}
}
Documentation: What are indexes?
Session : Querying : How to stream query results?
Important note: the Stream method will NOT track objects. If you change objects obtained from this method, SaveChanges() will not be aware of any change.
Other note: you may get the following exception if you do not specify the index to use.
InvalidOperationException: StreamQuery does not support querying dynamic indexes. It is designed to be used with large data-sets and is unlikely to return all data-set after 15 sec of indexing, like Query() does.

Couchdb views and many (thousands) document types

I'm studing CouchDB and I'm picturing a worst case scenario:
for each document type I need 3 view and this application can generate 10 thousands of document types.
With "document type" I mean the structure of the document.
After insertion of a new document, couchdb make 3*10K calls to view functions searching for right document type.
Is this true?
Is there a smart solution than make a database for each doc type?
Document example (assume that none documents have the same structure, in this example data is under different keys):
[
{
"_id":"1251888780.0",
"_rev":"1-582726400f3c9437259adef7888cbac0"
"type":'sensorX',
"value":{"ValueA":"123"}
},
{
"_id":"1251888780.0",
"_rev":"1-37259adef7888cbac06400f3c9458272"
"type":'sensorY',
"value":{"valueB":"456"}
},
{
"_id":"1251888780.0",
"_rev":"1-6400f3c945827237259adef7888cbac0"
"type":'sensorZ',
"value":{"valueC":"789"}
},
]
Views example (in this example only one per doc type)
"views":
{
"sensorX": {
"map": "function(doc) { if (doc.type == 'sensorX') emit(null, doc.valueA) }"
},
"sensorY": {
"map": "function(doc) { if (doc.type == 'sensorY') emit(null, doc.valueB) }"
},
"sensorZ": {
"map": "function(doc) { if (doc.type == 'sensorZ') emit(null, doc.valueC) }"
},
}
The results of the map() function in CouchDB is cached the first time you request the view for each new document. Let me explain with a quick illustration.
You insert 100 documents to CouchDB
You request the view. Now the 100 documents have the map() function run against them and the results cached.
You request the view again. The data is read from the indexed view data, no documents have to be re-mapped.
You insert 50 more documents
You request the view. The 50 new documents are mapped and merged into the index with the old 100 documents.
You request the view again. The data is read from the indexed view data, no documents have to be re-mapped.
I hope that makes sense. If you're concerned about a big load being generated when a user requests a view and lots of new documents have been added you could look at having your import process call the view (to re-map the new documents) and have the user request for the view include stale=ok.
The CouchDB book is a really good resource for information on CouchDB.
James has a great answer.
It looks like you are asking the question "what are the values of documents of type X?"
I think you can do that with one view:
function(doc) {
// _view/sensor_value
var val_names = { "sensorX": "valueA"
, "sensorY": "valueB"
, "sensorZ": "valueC"
};
var value_name = val_names[doc.type];
if(value_name) {
// e.g. "sensorX" -> "123"
// or "sensorZ" -> "789"
emit(doc.type, doc.value[value_name]);
}
}
Now, to get all values for sensorY, you query /db/_design/app/_view/sensor_value with a parameter ?key="sensorX". CouchDB will show all values for sensorX, which come from the document's value.valueA field. (For sensorY, it comes from value.valueB, etc.)
Future-proofing
If you might have new document types in the future, something more general might be better:
function(doc) {
if(doc.type && doc.value) {
emit(doc.type, doc.value);
}
}
That is very simple, and any document will work if it has a type and value field. Next, to get the valueA, valueB, etc. from the view, just do that on the client side.
If using the client is impossible, use a _list function.
function(head, req) {
// _list/sensor_val
//
start({'headers':{'Content-Type':'application/json'}});
// Updating this will *not* cause the map/reduce view to re-build.
var val_names = { "sensorX": "valueA"
, "sensorY": "valueB"
, "sensorZ": "valueC"
};
var row;
var doc_type, val_name, doc_val;
while(row = getRow()) {
doc_type = row.key;
val_name = val_names[doc_type];
doc_val = row.value[val_name];
send("Doc " + row.id + " is type " + doc_type + " and value " + doc_val);
}
}
Obviously use send() to send whichever format you prefer for the client (such as JSON).