I have a C# service that responds to clients periodically requesting an array of actions to perform and each action is stored in a RavenDB Action document with these properties where the last 2 properties are denormalized for performance:
Id (String - PK)
ClientId (String - FK)
RequestId (String - Unique Id for the request. We don't store a Request entity)
RequestDateTime (Date & Time - Date & time that request was made)
RequestDuration (Time Span - How long the request took to determine the list of actions)
I want to create an MR index that provides hourly request statistics per client so that I can see statistics for Client #1, 01/02/22 09:00-10:00 etc. I'm struggling to calculate AvgRequestDuration because the group contains duplicate RequestDuration(s) due to the data being denormalized. Obviously min & max are not affected with duplicates.
public class Result
{
public string ClientId { get; set; }
public DateTime PeriodStart { get; set; }
public TimeSpan MinRequestDuration { get; set; }
public TimeSpan MaxRequestDuration { get; set; }
public TimeSpan AvgRequestDuration { get; set; }
}
public ClientStatsByPeriodStartDateTime()
{
Map = action => from ac in actions
let period = TimeSpan.FromHours(1)
select new
{
ClientId = ac.ClientId,
PeriodStart = new DateTime(((ac.RequestDateTime.Ticks + period.Ticks - 1) / period.Ticks) * period.Ticks, DateTimeKind.Utc),
ac.RequestDuration
};
Reduce = results => from result in results
group result by new
{
result.ClientId,
result.PeriodStart
}
into agg
select new
{
ClientId = agg.Key.ClientId,
PeriodStart = agg.Key.PeriodStart,
AvgRequestDuration = agg.Avg(x => x.RequestDuration), // This is wrong
MinRequestDuration = agg.Min(x => x.RequestDuration),
MaxRequestDuration = agg.Max(x => x.RequestDuration)
};
}
Consider using the timeSeries feature to calculate avg, min & max.
Create a time series entry for each request.
The entry value can hold the duration.
You can then query for data at specific times, and get min,max,avg info for the values.
You can even index time series data.
This blog post can also be useful to start.
I've decided to normalize the structure and have a single document named Request that contains an array of the Action entity. The Duration property can then be stored against the Request document.
Related
I am still new to C# and I am struggling to find a solution to my problem. My SQL dapper query returns a table (based on my understanding though it is not really a table if it is IEnumerable unlike what I am use to working with ADO and recordsets) with three columns col1, col2, and col3 and has multiple rows. I need to loop through this query result for each row and test the values (ie, a foreach loop where I check row(0).field1=5, row(1).field1 = 5 for each row, etc) do what I need to do. This seems so basic but I all the dapper tutorials I see do not show examples for this and if they do they seem to utilize class objects rather than accessing the results directly (if thats even possible or do you have to map the results to a model?) My code is as follows:
String query = "exec dbo.storeProcedure #jsonData, #mainDocJSON, #supportingDocsJSON";
IEnumerable queryResult;
using (var connection = new SqlConnection(connectionString))
{
queryResult = connection.Query(query, new { jsonData = jsonData, mainDocJSON = mainDocJSON, supportingDocsJSON = supportingDocsJSON });
}
I also end up returning IEnumerable results from the controller this code resides in so I send it back to the user in JSON using the following.
return Ok(queryResult);
connection.Query return a IEnumerable, why dont we create a class to map the set from ? Dapper is a micro-ORM, but still... ORM.
For ex: Your table return 3 column Id, Name, CreatedDate.
// declare a class to map the result first
public class ResultHolderDto
{
public int Id { get; set; }
public string Name { get; set; }
public DateTime CreatedDate { get; set; }
}
// query somewhere
// This will return IEnumerable<ResultHolderDto>, feel free to play around as normal
var queryResult = await connection.QueryAsync<ResultHolderDto>(query, new { jsonData = jsonData, mainDocJSON = mainDocJSON, supportingDocsJSON = supportingDocsJSON });
foreach(var item in queryResult)
{
var col1Value = queryResult.Id;
var col2Value = queryResult.Name;
var col3Value = queryResult.CreatedDate;
// Then do something with col1Value, col2Value, col3Value...
}
My application has a requirement that is should be able to filter/search for Pairs by the Number of the related Contact.
A Pair always has a reference to a Contact stored, but the number of the contact is not, and will not, be stored in the reference. So I tried to create a custom index for this, because the Pair and Contact are stored in different collections.
A simplified example of the index looks like this.
public class Pairs_Search : AbstractMultiMapIndexCreationTask<Pairs_Search.Result>
{
public class Result
{
public string Id { get; set; }
public string Workspace { get; set; }
public ContactResult Contact { get; set; }
public bool HasContactDetails { get; set; }
}
public class ContactResult
{
public string Id { get; set; }
public string Name { get; set; }
public int Number { get; set; }
}
public Pairs_Search()
{
AddMap<Pair>(pairs => pairs
.Select(p => new
{
p.Id,
p.Workspace,
Contact = new
{
p.Contact.Id,
p.Contact.Name,
Number = 0
},
// Mark this items as WITHOUT contact details.
HasContactDetails = false,
}
)
);
AddMap<Contact>(contacts => contacts
.Select(c => new
{
Id = (string) null,
Workspace = (string) null,
Contact = new
{
c.Id,
Name = c.DisplayName,
c.Number
},
// Mark this items as WITH contact details.
HasContactDetails = true,
}
)
);
Reduce = results => results
// First group by the contact ID. This will
// create a group with 2 or more items. One with the contact
// details, and one or more with pair details.
// They are all marked by a boolean flag 'HasContactDetails'.
.GroupBy(x => x.Contact.Id)
// We are going to enrich each item in the current group, that is
// marked as 'HasContactDetails = false', with the contact number.
// We need that so that we can filter on it later.
.Select(group =>
group
.Select(i => new
{
i.Id,
i.Workspace,
Contact = new
{
i.Contact.Id,
i.Contact.Name,
// Does the current item have the contact details?
Number = i.HasContactDetails
// Yes, in this case we use the previously set contact number.
? i.Contact.Number
// No, find the item with the contact details and grab the number.
: group.Single(x => x.HasContactDetails).Contact.Number
},
// Pass on the flag that indicates wheter or not
// this item has the contact details. We are going
// to need it later.
i.HasContactDetails
}
)
// We don't need the items with the contact details
// anymore, so filter them out.
.Where(x => !x.HasContactDetails)
)
// Flatten all the small lists to one big list.
.SelectMany(x => x);
// Mark the following fields of the result as searchable.
Index(x => x.Contact.Number, FieldIndexing.Search);
}
}
I've setup a full example that reproduces the issues I am having. You can find the example here.
Creating the index works fine. Querying the index works fine also as it properly matched the pair and contact and enriched the index result with the number of the contact. But when I try to use a .Where() or .Search() on the nested Number property it fails to properly filter the result dataset from the index.
The index without any filtering works as you can see in below code example (also available in the full example).
private static async Task ThisOneWorks()
{
using (var session = Store.OpenAsyncSession())
{
var results = await session
.Query<Pairs_Search.Result, Pairs_Search>()
.ToListAsync();
LogResults("ThisOneWorks()", results);
}
// Output:
// ThisOneWorks(): Pair 'Harry Potter' with number '70'
// ThisOneWorks(): Pair 'Harry Potter' with number '70'
// ThisOneWorks(): Pair 'Hermione Granger' with number '71'
// ThisOneWorks(): Pair 'Albus Dumbledore' with number '72'
}
Filtering on a non-nested value also works (also available in the full example). As you can see it filters out the one with a different workspace.
private static async Task ThisOneWithWorkspaceFilterWorks()
{
using (var session = Store.OpenAsyncSession())
{
var results = await session
.Query<Pairs_Search.Result, Pairs_Search>()
.Where(x => x.Workspace == "hogwarts")
.ToListAsync();
LogResults("ThisOneWithWorkspaceFilterWorks()", results);
}
// Output:
// ThisOneWithWorkspaceFilterWorks(): Pair 'Harry Potter' with number '70'
// ThisOneWithWorkspaceFilterWorks(): Pair 'Harry Potter' with number '70'
// ThisOneWithWorkspaceFilterWorks(): Pair 'Hermione Granger' with number '71'
}
When I try to filter/search on the Workspace and Number properties I would expect two results that are related to the contact Harry Potter. But instead I just get an empty dataset back.
private static async Task ThisOneWithWorkspaceAndNumberFilterDoesntWork()
{
using (var session = Store.OpenAsyncSession())
{
var results = await session
.Query<Pairs_Search.Result, Pairs_Search>()
.Where(x => x.Workspace == "hogwarts")
.Where(x => x.Contact.Number == 70)
.ToListAsync();
LogResults("ThisOneWithWorkspaceAndNumberFilterDoesntWork()", results);
}
// Output:
// ThisOneWithWorkspaceAndNumberFilterDoesntWork(): EMPTY RESULTS!
}
Can anyone tell me what I am doing wrong here? Any help would be greatly appreciated!
The way to go about it is to store ContactResult in a different collection,
which is what is called a related document in this case,
and when you create the index then you 'Index the Related Document'
Learn from the demo example in:
https://demo.ravendb.net/demos/csharp/related-documents/index-related-documents
The example is for a basic map index but the principle is the same for Multi-Map.
Remove the public class ContactResult from the index class
and define the index with something like:
select new Result
{
....
Number = LoadDocument<Contact>(Pair.Contact).Number
....
}
Background
I'm attempting to use ServiceStack.OrmLite to grab some values (so I can cache them to run some processing against them).
I need to grab a combination of three values, and I have a custom SQL statement that will yield them (does the joins, etc.)
Because this will be a large list of combinations, I'd like to pass in some lists of values and use Sql.In to filter to only the results that have those values.
Specifics
I need to check whether an invoice is unique to a firm and another value (called ClaimLawsuitID here).
so have my poco:
public class FirmIDClaimLawsuitIDInvoiceNumberCombination
{
public string FirmID { get; set; }
public string ClaimLawsuitID { get; set; }
public string InvoiceNumber { get; set; }
}
and I have my SQL statement:
select tblDefenseInvoice.FirmID, tblDefInvClaimantDetail.ClaimLawsuitID, tblDefInvClaimantDetail.invoiceNumber
from tblDefenseInvoice
inner join tblDefInvClaimantDetail
on(tblDefenseInvoice.DefenseInvoiceID = tblDefInvClaimantDetail.DefenseInvoiceID)
I would like to run the following:
public List<FirmIDClaimLawsuitIDInvoiceNumberCombination> GetFirmIDClaimLawsuitIDInvoiceNumberCombinationsForExistingItems(IEnumerable<int> firmIds, IEnumerable<long> claimLawsuitIDs, IEnumerable<string> invoiceNumbers)
{
var sql = #"select tblDefenseInvoice.FirmID, tblDefInvClaimantDetail.ClaimLawsuitID, tblDefInvClaimantDetail.invoiceNumber
from tblDefenseInvoice
inner join tblDefInvClaimantDetail
on(tblDefenseInvoice.DefenseInvoiceID = tblDefInvClaimantDetail.DefenseInvoiceID)";
var ev = OrmLiteConfig.DialectProvider.ExpressionVisitor<tblClaimLawsuit>();
var firmFilter = PredicateBuilder.True<tblDefenseInvoice>();
var claimLawsuitFilter = PredicateBuilder.True<tblDefInvClaimantDetail>();
var invoiceNumberFilter = PredicateBuilder.True<tblDefInvClaimantDetail>();
firmFilter = x => Sql.In(x.FirmID, firmIds);
claimLawsuitFilter = x => Sql.In(x.ClaimLawsuitID, claimLawsuitIDs);
invoiceNumberFilter = x => Sql.In(x.InvoiceNumber, invoiceNumbers);
ev.Select(sql);
ev.Where(firmFilter);
ev.Where(claimLawsuitFilter);
ev.Where(invoiceNumberFilter);
return dal.DB.Select<FirmIDClaimLawsuitIDInvoiceNumberCombination>(ev.ToSelectStatement());
}
Question
Is this possible to achieve this way?
Is there some other way of achieving this within ServiceStack's OrmLite that is cleaner and I'm unaware of?
Since I was selecting to a POCO, I simply needed to add the filters based on that POCO.
The following worked just fine:
public List<FirmIDClaimLawsuitIDInvoiceNumberCombination>
GetFirmIDClaimLawsuitIDInvoiceNumberCombinationsForExistingItems(
IEnumerable<long> firmIds,
IEnumerable<long> claimLawsuitIDs)
{
var sql = #"select tblDefenseInvoice.FirmID, tblDefInvClaimantDetail.ClaimLawsuitID, tblDefInvClaimantDetail.invoiceNumber
from tblDefenseInvoice
inner join tblDefInvClaimantDetail
on(tblDefenseInvoice.DefenseInvoiceID = tblDefInvClaimantDetail.DefenseInvoiceID)";
var ev = OrmLiteConfig.DialectProvider.ExpressionVisitor<FirmIDClaimLawsuitIDInvoiceNumberCombination>();
var firmFilter = PredicateBuilder.True<FirmIDClaimLawsuitIDInvoiceNumberCombination>();
var claimLawsuitFilter = PredicateBuilder.True<FirmIDClaimLawsuitIDInvoiceNumberCombination>();
firmFilter = x => Sql.In(x.FirmID, firmIds);
claimLawsuitFilter = x => Sql.In(x.ClaimLawsuitID, claimLawsuitIDs);
ev.Select(sql);
ev.Where(firmFilter);
ev.Where(claimLawsuitFilter);
return dal.DB.Select<FirmIDClaimLawsuitIDInvoiceNumberCombination>(ev.ToSelectStatement());
}
If I want single primitive value like int, string, float etc I can do like this.
using (var db = new SQLiteConnection(DbPath))
{
double i = db.CreateCommand("select salary from PersonMaster where personId = ?", 9).ExecuteScalar<double>();
}
If I try to return whole object of person master i.e. single row the below code returns null
using (var db = new SQLiteConnection(DbPath))
{
PersonMaster objPersonMaster = db.CreateCommand("select * from PersonMaster where personId = ?", 9).ExecuteScalar<PersonMaster>();
}
I compulsory have to use this
using (var db = new SQLiteConnection(DbPath))
{
List<PersonMaster> lstPersonMaster = db.Query<PersonMaster>("select * from PersonMaster where personId = ?", 9);
PersonMaster objPersonMaster = lstPersonMaster.First();
}
Is there any way to get single row as object, rather than dealing with List<T>
I am assuming you are using SQLite-Net. If that is the case you can use the Find method. Find gets an object using its primary key.
The personId field needs the PrimaryKey attribute as follows:
public class PersonMaster
{
[PrimaryKey]
public int personId { get; set; }
public string name { get; set; }
public decimal salary { get; set; }
}
Then you can use Find like so:
// get person 9
PersonMaster person9 = db.Find<PersonMaster>(9);
You can either use "TOP(1) yourquery" or "yourquery LIMIT 1"
if you just want to get rid of the list step, you can do it directly:
PersonMaster objPersonMaster = db.Query<PersonMaster>("select * from PersonMaster where personId = ?", 9).First();
Of course you have to deal with a possible exception in case nothing is returned, but you should do this also with the list approach anyway.
Following this question
I have the following document structure:
Game
- Id
- Teams
- Team 1
- list of players
- Team 2
- list of players
- Events
- Event 1
- type: Pass
- teamId
- PlayerId
- Event 2
- type: Goal
- teamId
- PlayerId
How do I build an index that gives me all the events for a player for a given game?
Here is how far I got and RavenDB says it can't understand my query?
public class Games_PlayerEvents : AbstractIndexCreationTask<Game, Games_PlayerEvents.ReduceResult>
{
public class ReduceResult
{
public string PlayerId { get; set; }
public IEnumerable<GameEvent> Events { get; set; }
}
public Games_PlayerEvents()
{
Map = games => from game in games
select new
{
PlayerId = "",
Events = game.Events
};
Reduce = results => from result in results
from #event in result.Events
group #event by #event.PlayerId into playerEvents
select new
{
PlayerId = playerEvents.Key,
Events = playerEvents.Select(g => g)
};
}
}
Marto, in case you really want to model it this way, you don't need to create an index, as you can load the whole game document and do the rest using standard linq-to-objects aggregations.
Anyway, it sounds as if you want to have Game, Player and Event as independant documents as these are your aggregate roots or in other words - they have a meaning on their own.