LINQ or Navigation Properties command to retrieve 1 to many data - sql

Question

I am looking for help with a LINQ SQL query please.
I have a blazor application that gets data from an Azure SQL database. I am seeking to get a dataset from the database for linking to a datagrid, where each row is a record from the main table joined with a record from the second table. The second table has millions of records, it needs to join one record which has the same key (securityId) and with the date being the record with the nominated date, or with the last date before the nominated date.
Because of the size of the 2nd file, I need an efficient query. Currently I am using the following, but I believe there must be more efficient ways to do it without the lag. Also tried Navigation Properties but couldn't get to work
reviewdateS is the date that I want the 2nd record to match or be the latest date prior to that date
result = (from cmpn in _ctx.MstarCompanies
join prcs in _ctx.MstarPrices
on cmpn.securityId equals prcs.securityId into cs
from c in cs.DefaultIfEmpty()
where c.date01 == reviewDateS
select new ClsMarketPrices { })
Following are the 3 relevant classes. ClsMarketPrices does not relate to a database table, it is simple a class that combines the other 2 classes which may not be necessary but with my limited knowledge it is how it is working.
_ctx is a repository that links to the data context.
public MySQLRepositories(ApplicationDbContext ctx)
{
_ctx = ctx;
}
public class ClsMarket
{
[Key]
public int CompanyId { get; set; } = 0;
public string securityId { get; set; } = "";
public string companyName { get; set; } = "";
public string mic { get; set; } = "";
public string currency { get; set; } = "";
[ForeignKey("securityId")]
public virtual ICollection<ClsPrices> Prices { get; set; }
}
public class ClsMarketPrices
{
[Key]
public int CompanyId { get; set; } = 0;
public string companyName { get; set; } = "";
public string period { get; set; } = "";
public string mic { get; set; } = "";
}
public class ClsPrices
{
[Key]
public int PricesId { get; set; }
[ForeignKey("securityId")]
public string securityId { get; set; } = "";
public string mic { get; set; } = "";
public string date01 { get; set; } = "";
public virtual ClsMarket ClsMarket {get; set;}
}

Answer 1

I want to get a record from the 1st file joined with a record from the 2nd file where that record from the 2nd file has a date equal to or the last before the nominated date.
So we are talking about files, not a database! This is important, because this means that your local process will execute the LINQ, not a database management system. In other words: the LINQ will be IEnumerable, not IQueryable.
This is important, because as Enumerable, you will be able to define your own LINQ extension methods.
Although you supplied an enormous amount of irrelevant properties, you forgot to give us the most important things: you were talking about two files, you told us that you have two classes with a one-to-many relation, but you gave us three classes. Which ones do have the relation that you are talking about?
I think that every object of ClsMarketPrices has zero or more ClsPrices, and that every ClsPrice is one of the prices of a ClsMarketPrices, namely the ClsMarketPrices that the foreign key SecurityId (rather confusing name) refers to.
First of all, let's assume you already have procedures to read the two sequences from your files. And of course, these procedures won't read more than needed (so don't read the whole file if you will only use the first ClsMarket). I assume you already know how to do that:
IEnumerable<ClsMarketPrices> ReadMarketPrices();
IEnumerable<ClsPrices> ReadPrices();
So you've go a DateTime reviewDate. Every MarketPrice has zero or more Prices. Every Price has a DateTime property DateStamp. You want for every MarketPrice the Price that has the largest value for DateStamp that is smaller or equal to reviewDate.
If a MarketPrice doesn't have such a Prices, for instance because it doesn't have a Price at all, or all its Prices have a DateStamp larger than reviewDate, you want a value null.
You didn't say what you want if a MarketPrice has several Prices with equal largest DateStamp <= reviewDate. I assume that you don't care which one is selected.
The straighforward LINQ method would be to use GroupJoin, Where, Orderby and FirstOrDefault:
DateTime reviewDate = ...
IEnumerable<ClsMarketPrices> marketPricess = ReadMarketPrices();
IEnumerable<ClsPrices> prices = ReadPrices().Where(price => price.DateStamp <= reviewDate);
// GroupJoin marketPrices with prices:
var result = markets.GroupJoin(prices,
marketPrice => marketPrice.CompanyId, // from every MarketPrice take the primary key
price => price.CompanyId, // from every price take the foreign key to its market
// parameter resultSelector: from every market, with its zero or more matching prices
// make one new:
(marketPrice, pricesOfThisMarketPrice) => new
{
// select the marketPrice properties that you plan to use:
Id = marketPrice.CompanyId,
Name = ...
...
// from all prices of this marketPrice, take the one with the largest DateStamp
// we know there are no marketPrices with a DataStamp larger than reviewData
LatestPrice = pricesOfThisMarketPrice.OrderbyDescending(price => price.DateStamp)
.Select(price => new
{
// Select the price properties you plan to use;
Id = price.PricesId,
Date = price.DateStamp,
...
})
.FirstOrDefault(),
});
The problem is: this must be done efficiently, because you have an immense amount of Markets and MarketPrices.
Althoug we already limited the amount of prices to sort by removing the prices that are after reviewDate, it is still a waste to order all Dates if you will only be using the first one.
We can optimize this, by using Aggregate for pricesOfThisMarketPrice. This will assert that pricesOfThisMarketPrice will be enumerated only once.
Side remarks: Aggregate only works on IEnumerable, not on IQueryable, so it won't work on a database. Furthermore, pricesOfThisMarketPrice might be an empty sequence; we have to take care of that.
LatestPrice = pricesOfThisMarketPrice.Any() ?
pricesOfThisMarketPrice.Aggregate(
// select the one with the largest value of DateStamp:
(latestPrice, nextPrice) => nextPrice.DateStamp >= latesPrice.DateStamp) ? nextPrice : latestPrice)
// do not do the aggregate if there are no prices at all:
: null,
Although this Aggregate is more efficient than OrderBy, your second sequence will still be enumerated more than once. See the source code of Enumerable.GroupJoin.
If you really want to enumerate your second source once, and limit the number of enumerations of the first source, consider to create an extension method. This way you can use it as any LINQ method. If you are not familiar with extension methods, see extension methods demystified.
You can create an extension method for your ClsPrices and ClsPrice, however, if you think you will need to "find the largest element that belongs to another element" more often, why not create a generic method, just like LINQ does.
Below I create the most extensive extension method, one with a resultSelector and equalityComparers. If you will use standard equality, consider to add an extension method without these comparers and let this extension method call the other extension method with null value for the comparers.
For examples about the overloads with and without equality comparers see several LINQ methods, like ToDictionary: there is a method without a comparer and one with a comparer. This first one calls the second one with null value for comparer.
I will use baby steps, so you can understand what happens.
This can slightly be optimized.
The most important thing is that you will enumerate your largest collection only once.
IEnumerable<TResult> TakeLargestItem<T1, T2, TKey, Tproperty, TResult>(
this IEnumerable<T1> t1Sequence,
IEnumerable<T2> t2Sequence,
// Select primary and foreign key:
Func<T1, TKey> t1KeySelector,
Func<T2, TKey> t2KeySelector,
// Select the property of T2 of which you want the largest element
Func<T2, TProperty> propertySelector,
// The largest element must be <= propertyLimit:
TProperty propertyLimit,
// From T1 and the largest T2 create one TResult
Func<T1, T2, TResult> resultSelector,
// equality comparer to compare equality of primary and foreign key
IEqualityComparer<TKey> keyComparer,
// comparer to find the largest property value
IComparer<TProperty> propertyComparer)
{
// TODO: invent a property method name
// TODO: decide what to do if null input
// if no comparers provided, use the default comparers:
if (keyComparer == null) keyComparer = EqualityComparer<TKey>.Default;
if (propertyComparer == null) propertyComparer = Comparer<TProperty>.Default;
// TODO: implement
}
The implementation is straightforward:
put all T1 in a dictionary t1Key as key, {T1, T2} as value, keyComparer as comparer
then enumerate T2 only once.
check if the property <= propertyLimit,
if so, search in the dictionary for the {T1, T2} combination with the same key
check if the current t2Item is larger than the T2 in the {T1, T2} combination
if so: replace
We need an internal class:
class DictionaryValue
{
public T1 T1 {get; set;}
public T2 T2 {get; set;}
}
The code:
IDictionary<TKey, DictionaryValue> t1Dict = t1Sequence.ToDictionary(
t1 -> t1KeySelector(t1),
t1 => new DictionaryValue {T1 = t1, T2 = (T2)null },
keyComparer);
The enumeration of t2Sequence:
foreach (T2 t2 in t2Sequence)
{
// check if the property is <= propertyLimit
TProperty property = propertySelector(t2);
if (propertyComparer.Compare(property, propertyLimit) < 0)
{
// find the T1 that belongs to this T2:
TKey key = keySelector(t2);
if (t1Dict.TryGetValue(key, out DictionaryValue largestValue))
{
// there is a DictionaryValue with the same key
// is it null? then t2 is the largest
// if not null: get the property of the largest value and use the
// propertyComparer to see which one of them is the largest
if (largestValue.T2 == null)
{
largestValue.T2 = t2;
}
else
{
TProperty largestProperty = propertySelector(largestValue.T2);
if (propertyComparer.Compare(property, largestProperty) > 0)
{
// t2 has a larger property than the largestValue: replace
largestValue.T2 = t2,
}
}
}
}
}
So for every t1, we have found the largest t2 that has a property <= propertyLimit.
Use the resultSelector to create the results.
IEnumerable<TResult> result = t1Dict.Values.Select(
t1WithLargestT2 => resultSelector(t1WithLargestT2.T1, t1WithLargestT2.T2));
return result;