Good morning,
I have a collection with a huge number of documents, and what I want to do is group by a certain field and put the results into a new collection.
I know how to do this in MongoDB, and it works perfect:
db.collection_1.aggregate([
{ $group : {_id : "$field_1" } },
{ $out : "collection_group" } ])
I've found how to do this in C++ in the MongoDB documentation. In this case it executes a sum after grouping, I want to insert the result in a new collection, but I supose I would only have to change "$sum" for "$out" :
var collection = _database.GetCollection<BsonDocument>("collection_1");
var aggregate = collection.Aggregate().Group(new BsonDocument { { "_id", "$field_1" },
{ "count", new BsonDocument("$sum", 1) } });
The thing is that I can't find examples in vb .net. In vb .net, I've tried for example this to group:
Dim collection_act = db.GetCollection(Of BsonDocument)("collection_1")
collection_act.Group(New GroupArgs("_id", "$field_1"))
But it causes an error because the GroupArgs cannot be defined like this.
I tried to use the Aggregate method too, but I have the same problem defining the AggregateArgs, and I can't find how to define it.
The most interesting thing would be to create a pipeline and then add commands to it, because after doing this I have to remove documents based on these group.
Any help will be very appreciate.
Thanks.
Related
NOTE: If you do know that the below is not possible, this information is just as valuable.
Im checking out HotChocolate and Ive looked into the dynamic schemas
I have taken the code in your Github-example This works ok. I have extended our Product-type like so:
//Same json as in your Github-sample, new properties are "color" and some more
foreach (var field in type.GetProperty("fields").EnumerateArray())
{
string name = field.GetString();
typeDefinition.Fields.Add(new ObjectFieldDefinition(field.GetString()!, type: TypeReference.Parse("String"), pureResolver: ctx =>
{
var product = ctx.Parent<Product>();
return product.SubTypeSpecific.ContainsKey(name) ? product.SubTypeSpecific[name] : null;
}));
}
now I have "moved out" dynamic properties from a Dictionary (That is a sub-object in my documentDb) to own properties. Works well.
BUT:
How can I extend my ProductFilter in the same fashion?
I would like to extend my current Product-filter so its possible to search on the new dynamic property "color"
getProducts(where : color : {eq :"blue" }}) {...}
I can create new FilterInputDefinition, but not extend existing filter (because there is no FilterInputTypeExtensions.CreateUnsafe()
If I manage to create the new filter, is there any way to update the IQueryable-generation so that the inputed color ("blue")
So the query to my CosmosDb will be created automatically?
Many thanks in advance.
Lets say there is an array defined as sub document.
User
first_name:"blah",
last_name: "blah",
scores [{score: 12},
{score: 13},
{score: 9},
{score: 14}]
Say I want to loop through this sub array inside the update statement and check if I need to add a new score. New score could be 8 or 16 if its 8 then nothing should be added, however if its 16 it should be added to the end of the list.
Is there is anyway to define javascript server side function and use it inside update statement, or there are other way of doing something like that? The important thing that operation has to be atomic.
Using php driver
public bool|array MongoCollection::update ( array $criteria , array $new_object [, array $options = array() ] )
How do I specify criteria to match those conditions that I have listed?
You can create a server-side javascript function, however the creators of mongo do not recommend this:
We do not recommend using server-side stored functions if possible.
If you need atomicity with complex logic, one approach with is to lock the document before working with it. You can achieve this with the findAndModify function. The basic gist of this is that you try to find your document as you normally would, with the additional criteria that locked = false. The modify part of this function would set locked = true.
db.people.findAndModify({
query: { name: "Andy", locked: false },
update: { locked: true },
});
You now are free to work with the document "atomically" When you write the changes back to your database, you set locked = false. Make sure you wrap your block in a try/catch/finally and ensure the document is unlocked in the finally block.
Note that if your operation is very simple (simply need to add a field to your list), then using findAndModify alone to update the document fields may be enough for what you need.
Edit: If you really want to use server-side functions here's an example from the PHP mongo documentation:
$response = $db->execute("function(x) { return x;}", array("input param"));
Here's how you could persist a function for later use (through the mongo shell)
db.system.js.save(
{ _id: "echoFunction",
value : function(x) { return x; }
}
)
Then later in PHP:
$response = $db->execute("echoFunction( 'test' )");
var_dump($response);
Edit2:
Your update function is much easier than I first understood. You simply need the following:
$db->update(
array("first_name" => "blah"),
array(
"$push" => array(
"scores" => array("score" => 16)
)
)
);
Edit2: Access your collection from inside a function
function push_score(id, score) {
//your logic here
//...
db.your_collection.update({_id: ObjectId(id), $push: {scores: {score: score}}})
}
I am new to RavenDB and could really use some help.
I have a collection of ~20M documents, and I need to add a key to each document. The challenge is that the value of the key needs to be derived from another document.
For instance, given the following document:
{
"Name" : "001A"
"Date" : "09-09-2013T00:00:00.0000000"
"Related" : [
"002B",
"003B"
]
}
The goal is to add a key that holds the dates for the related documents, i.e. 002B and 003B, by looking up the related documents in the collection and returning their date. E.g.:
{
"Name" : "001A"
"Date" : "09-09-2013T00:00:00.0000000"
"Related" : [
"002B",
"003B"
]
"RelatedDates" : [
"08-10-2013T00:00:00.0000000",
"08-15-2013T00:00:00.0000000"
]
}
I realize that I'm trying to treat the collection somewhat like a relational database, but this is the form that my data is in to begin with. I would prefer not to put everything into a relational dataset first in order to structure the data for RavenDB.
I first tried doing this on the client side, by paging through the collection and updating the records. However, I quickly reach the maximum number of request for the session.
I then tried patching on the server side with JavaScript, but I'm not sure if this is possible.
At this point I would greatly appreciate some strategic guidance on the right way to approach this problem, as well as, more tactical guidance on how to implement it.
The recommended way of doing this is via a Console application that loops thru all your records, similar to what you have already done but in a way that pages the data so you dont hit the maximum number of requests per session.
See this example from the ravendb source code example application:
you need to do something like this:
using (var store = new DocumentStore { ConnectionStringName = "RavenDB" }.Initialize())
{
int start = 0;
while (true)
{
using (var session = store.OpenSession())
{
var posts = session.Query<Post>()
.OrderBy(x => x.CreatedAt)
.Include(x => x.CommentsId)
.Skip(start)
.Take(128)
.ToList();
if (posts.Count == 0)
break;
foreach (var post in posts)
{
session.Load<PostComments>(post.CommentsId).Post = new PostComments.PostReference
{
Id = post.Id,
PublishAt = post.PublishAt
};
}
session.SaveChanges();
start += posts.Count;
Console.WriteLine("Migrated {0}", start);
}
}
}
I've done this sort of thing with about ~1.5M records and it wasnt exactly quick to do the migration. If your records are small then you can just Load<> and SaveChanges on each one as from experience programmatically patching the documents did not speed things up materially
As a side note, the ravendb google groups is very active if you want to ask specifically about doing this from the studio
I am trying to save (RavenDB build 960) the names and values of form data items passed into a Nancy Module via its built in Request.Form.
If I save a straightforward instance of a dynamic object (with test properties and values) then everything works and both the property names and values are saved. However, if I use Nancy's Request.Form then only the dynamic property names are saved.
I understand that I will have to deal with further issues to do with restoring the correct types when retrieving the dynamic data (RavenJObjects etc) but for now, I want to solve the problem of saving the dynamic names / values in the first place.
Here is the entire test request and code:
Fiddler Request (PUT)
Nancy Module
Put["/report/{name}/add"] = parameters =>
{
reportService.AddTestDynamic(Db, parameters.name, Request.Form);
return HttpStatusCode.Created;
};
Service
public void AddTestDynamic(IDocumentSession db, string name, dynamic data)
{
var testDynamic = new TestDynamic
{
Name = name,
Data = data
};
db.Store(testDynamic);
db.SaveChanges();
}
TestDynamic Class
public class TestDynamic
{
public string Name;
public dynamic Data;
}
Dynamic contents of Request.Form at runtime
Resulting RavenDB Document
{
"Name": "test",
"Data": [
"username",
"age"
]
}
Note: The type of the Request.Form is Nancy.DynamicDictionary. I think this may be the problem since it inherits from IEnumerable<string> and not the expected IEnumerable<string, object>. I think that RavenDB is enumerating the DynamicDictionary and only getting back the dynamic member-names rather than the member name / value pairs.
Can anybody tell me how or whether I can treat the Request.Form as a dynamic object with respect to saving it to RavenDB? If possible I want to avoid any hand-crafted enumeration of DynamicDictionary to build a dynamic instance so that RavenDB can serialise correctly.
Thank You
Edit 1 #Ayende
The DynamicDictionary appears to implement the GetDynamicMemberNames() method:
Taking a look at the code on GitHub reveals the following implementation:
public override IEnumerable<string> GetDynamicMemberNames()
{
return dictionary.Keys;
}
Is this what you would expect to see here?
Edit 2 #TheCodeJunkie
Thanks for the code update. To test this I have:
Created a local clone of the NancyFx/Nancy master branch from
GitHub
Added the Nancy.csproj to my solution and referenced the project
Run the same test as above
RavenDB Document from new DynamicDictionary
{
"Name": "test",
"Data": {
"$type": "Nancy.DynamicDictionary, Nancy",
"username": {},
"age": {}
}
}
You can see that the resulting document is an improvement. The DynamicDictionary type information is now being correctly picked up by RavenDB and whilst the dynamic property-names are correctly serialized, unfortunately the dynamic property-values are not.
The image below shows the new look DynamicDictionary in action. It all looks fine to me, the new Dictionary interface is clearly visible. The only thing I noticed was that the dynamic 'Results view' (as opposed to the 'Dynamic view') in the debugger, shows just the property-names and not their values. The 'Dynamic view' shows both as before (see image above).
Contents of DynamicDictionary at run time
biofractal,
The problem is the DynamicDictionary, in JSON, types can be either objects or lists ,they can't be both.
And for dynamic object serialization, we rely on the implementation of GetDynamicMemberNames() to get the properties, and I assume that is isn't there.
I am developing a game where I'd like to have multiple scripts that all implement the same structure. Each script would need to be run in its own scope so that code doesn't overlap other scripts. For example:
structure.js
function OnInit() {
// Define resources to load, collision vars, etc.
}
function OnLoop() {
// Every loop
}
function ClickEvent() {
// Someone clicked me
}
// Other fun functions
Now, lets say I have: "BadGuy.js", "ReallyReallyBadGuy.js", "OtherBadGuy.js" - They all look like the above in terms of structure. Within the game whenever an event takes place, I'd like to invoke the appropriate function.
The problem comes down to efficiency and speed. I found a working solution by creating an engine for each script instance (using getEngineByName), but that just doesn't seem ideal to me.
If there isn't a better solution, I'll probably resort to each script having its own unique class / function names. I.e.
BadGuy.js
var BadGuy = new Object();
BadGuy.ClickEvent = function() {
}
I don't think you need to create a new ScriptEngine for every "Guy". You can manage them all in one engine. So with advance apologies for butchering you game scenario.....
Get one instance of the Rhino engine.
Issue eval(script) statements to add new JS Objects to the engine, along with the different behaviours (or functions) that you want these Objects to support.
You have a couple of different choices for invoking against each one, but as long as each "guy" has a unique name, you can always reference them by name and invoke a named method against it.
For more performance sensitive operations (perhaps some sort of round based event loop) you can precompile a script in the same engine which can then be executed without having to re-evaluate the source.
Here's a sample I wrote in Groovy.
import javax.script.*;
sem = new ScriptEngineManager();
engine = sem.getEngineByExtension("js");
engine.getBindings(ScriptContext.ENGINE_SCOPE).put("out", System.out);
eventLoop = "for(guy in allGuys) { out.println(allGuys[guy].Action(action)); }; "
engine.eval("var allGuys = []");
engine.eval("var BadGuy = new Object(); allGuys.push(BadGuy); BadGuy.ClickEvent = function() { return 'I am a BadGuy' }; BadGuy.Action = function(activity) { return 'I am doing ' + activity + ' in a BAD way' }");
engine.eval("var GoodGuy = new Object(); allGuys.push(GoodGuy); GoodGuy.ClickEvent = function() { return 'I am a GoodGuy' }; GoodGuy.Action = function(activity) { return 'I am doing ' + activity + ' in a GOOD way' }");
CompiledScript executeEvents = engine.compile(eventLoop);
println engine.invokeMethod(engine.get("BadGuy"), "ClickEvent");
println engine.invokeMethod(engine.get("GoodGuy"), "ClickEvent");
engine.getBindings(ScriptContext.ENGINE_SCOPE).put("action", "knitting");
executeEvents.eval();