RavenDB StartsWith Variable on left-hand-side of Query - ravendb

I have a collection of docs with a Url property that looks like:
{ Url: www.test.com }
{ Url: www.test.com/abc }
{ Url: www.test.com/abc/xyz }
I want to query the docs with www.test.com/abc such that I am returned all the documents that match the url on everything up-to-and-including the query url. i.e. the results would return:
{ Url: www.test.com }
{ Url: www.test.com/abc }
Below is the query I have written to accomplish this. As you can see, it relies on the 'Starts With' to be performed on the queryTerm, as opposed to being performed on a property of the document (the left-hand-side of the query).
var queryTerm = "www.test.com/abc";
pages = session.Query<MyDocument>()
.Where(x => queryTerm.StartsWith(x.Url));
This does not work in Raven; I get an 'Expression type not supported' error. Any ideas?

Raven doesn't appear to support that in its expression. Interestingly, the query x => queryTerm.StartsWith(x.Url) can be exploded into something along the lines of:
x => x.Url == queryTerm ||
x.Url == queryTerm.Remove(queryTerm.Length - 1) ||
x.Url == queryTerm.Remove(queryTerm.Length - 2) ||
x.Url == queryTerm.Remove(queryTerm.Length - 3) ||
x.Url == queryTerm.Remove(queryTerm.Length - 4) ||
etc.
Which can be codified as:
var terms = Enumerable.Range(0, queryTerm.Length)
.Skip(1)
.Select(i => queryTerm.Remove(i));
pages = session.Query<MyDocument>()
.Where(x => terms.Contains(x.Url));
Except Raven doesn't supports Contains() like that either. So I coded this up:
using System.Collections.Generic;
using System.Linq;
using NUnit.Framework;
using Raven.Client;
using Raven.Client.Embedded;
using Raven.Client.Linq;
namespace RavenTest
{
[TestFixture]
public class HowToDoBackwardsStartsWithInRaven
{
private readonly string[] _testUrls = new[]
{
"www.bad.com",
"www.test.com",
"www.test.com/abc",
"www.test.com/abc/xyz"
};
private readonly string _queryTerm = "www.test.com/abc";
private readonly string[] _expectedResults = new[]
{
"www.test.com",
"www.test.com/abc"
};
[Test]
public void FailsWithContains()
{
List<MyDocument> results;
using (var store = InitStore())
{
LoadTestData(store);
using (var session = store.OpenSession())
{
var terms = GetAllStartingSubstrings(_queryTerm);
results = session.Query<MyDocument>()
.Where(x => terms.Contains(x.Url))
.ToList();
}
}
Assert.That(results.Select(p => p.Url), Is.EquivalentTo(_expectedResults));
}
[Test]
public void SucceedsWithSuperWhere()
{
List<MyDocument> results;
using (var store = InitStore())
{
LoadTestData(store);
using (var session = store.OpenSession())
{
var terms = GetAllStartingSubstrings(_queryTerm);
var query = session.Query<MyDocument>()
.Where(x => x.Url.Length <= _queryTerm.Length);
foreach (var term in terms)
query = query.Where(x => x.Url == term ||
x.Url.Length != term.Length);
results = query.ToList();
}
}
Assert.That(results.Select(p => p.Url), Is.EquivalentTo(_expectedResults));
}
private static IDocumentStore InitStore()
{
var store = new EmbeddableDocumentStore
{
RunInMemory = true,
};
return store.Initialize();
}
private static string[] GetAllStartingSubstrings(string queryTerm)
{
return Enumerable.Range(0, queryTerm.Length)
.Skip(1)
.Select(i => queryTerm.Remove(i))
.ToArray();
}
private void LoadTestData(IDocumentStore store)
{
using (var session = store.OpenSession())
{
foreach (var testUrl in _testUrls)
session.Store(new MyDocument {Url = testUrl});
session.SaveChanges();
}
}
public class MyDocument
{
public string Id { get; set; }
public string Url { get; set; }
}
}
}
Now... I have no idea how efficient the indexing will be on that but the test does pass.

Related

Is there a way to improve the code? .Net Core 6.0 and Cosmos Database with SQL API

It's my first time using .Net Core 6.0 WebAPI and Cosmos Database with SQL API, so I'm concerned that I've done things correctly.
I have used /address/zipcode as a partition key in the below sample
public class FamilyService : IFamilyService
{
public FamilyService(CosmosClient dbClient, string databaseName, string containerName)
{
this._container = dbClient.GetContainer(databaseName, containerName);
}
public Task<Family> GetFamilyDataAsync(string id, string zipcode)
{
return Task.FromResult(_container.GetItemLinqQueryable<Family>(allowSynchronousQueryExecution: true)
.Where(p => p.Id == id && p.Address.ZipCode == zipcode)
.ToList().First());
}
public async Task AddFamilyDataAsync(Family family)
{
await this._container.CreateItemAsync<Family>(family, new PartitionKey(family.Address.ZipCode));
}
public async Task UpdateFamilyDataAsync(Family family)
{
await this._container.ReplaceItemAsync<Family>(family, family.Id, new PartitionKey(family.Address.ZipCode));
}
public async Task DeleteFamilyDataAsync(string id, string zipcode)
{
await this._container.DeleteItemAsync<Family>(id, new PartitionKey(zipcode));
}
private async Task<bool> GetFamilyDataFromId(string id, string zipcode)
{
string query = $"select * from c where c.id=#familyId";
QueryDefinition queryDefinition = new QueryDefinition(query).WithParameter("#familyId", id);
List<Family> familyResults = new List<Family>();
FeedIterator streamResultSet = _container.GetItemQueryStreamIterator(
queryDefinition,
requestOptions: new QueryRequestOptions()
{
PartitionKey = new PartitionKey(zipcode),
MaxItemCount = 10,
MaxConcurrency = 1
});
while (streamResultSet.HasMoreResults)
{
using (ResponseMessage responseMessage = await streamResultSet.ReadNextAsync())
{
if (responseMessage.IsSuccessStatusCode)
{
dynamic streamResponse = FromStream<dynamic>(responseMessage.Content);
List<Family> familyResult = streamResponse.Documents.ToObject<List<Family>>();
familyResults.AddRange(familyResult);
}
else
{
return false;
}
}
}
if (familyResults != null && familyResults.Count > 0)
{
return true;
}
return familyResults;
}
Especially I want to focus more on those methods that retrieves the data from the database.
Update#1 : I have updated the code as suggested by #Mark Brown.
public class FamilyService : IFamilyService
{
private Container _container;
private static readonly JsonSerializer Serializer = new JsonSerializer();
public FamilyService(CosmosClient dbClient, string databaseName, string containerName)
{
this._container = dbClient.GetContainer(databaseName, containerName);
}
public async Task<Family> GetFamilyDataAsync(string id, string zipCode)
{
return await _container.ReadItemAsync<Family>(id, new PartitionKey(zipCode));
}
public async Task<List<Family>> GetAllFamilyDataAsync(string zipCode)
{
string query = $"SELECT * FROM Family";
QueryDefinition queryDefinition = new QueryDefinition(query);
List<Family> familyResults = new List<Family>();
FeedIterator<Family> feed = _container.GetItemQueryIterator<Family>(
queryDefinition,
requestOptions: new QueryRequestOptions()
{
PartitionKey = new PartitionKey(zipCode),
MaxItemCount = 10,
MaxConcurrency = 1
});
while (feed.HasMoreResults)
{
FeedResponse<Family> response = await feed.ReadNextAsync();
foreach (Family item in response)
{
familyResults.Add(item);
}
}
return familyResults;
}
public async Task<List<Family>> GetAllFamilyDataByLastNameAsync(string lastName, string zipCode)
{
string query = $"SELECT * FROM Family where Family.lastName = #lastName";
QueryDefinition queryDefinition = new QueryDefinition(query).WithParameter("#lastName", lastName);
List<Family> familyResults = new List<Family>();
FeedIterator<Family> feed = _container.GetItemQueryIterator<Family>(
queryDefinition,
requestOptions: new QueryRequestOptions()
{
PartitionKey = new PartitionKey(zipCode),
MaxItemCount = 10,
MaxConcurrency = 1
});
while (feed.HasMoreResults)
{
FeedResponse<Family> response = await feed.ReadNextAsync();
foreach (Family item in response)
{
familyResults.Add(item);
}
}
return familyResults;
}
public async Task<List<Family>> GetAllFamilyDataPaginatedAsync(int pageNumber, string zipCode)
{
int pageSize = 2;
int itemsToSkip = (pageSize * pageNumber) - pageSize;
QueryRequestOptions options = new QueryRequestOptions { MaxItemCount = pageSize };
string continuationToken = null;
List<Family> familyResults = new List<Family>();
IQueryable<Family> query = this._container
.GetItemLinqQueryable<Family>(false, continuationToken, options)
.Where(i => i.Address.ZipCode == zipCode)
.Skip(itemsToSkip);
using (FeedIterator<Family> iterator = query.ToFeedIterator<Family>())
{
FeedResponse<Family> feedResponse = await iterator.ReadNextAsync();
foreach (Family item in feedResponse)
{
familyResults.Add(item);
}
}
return familyResults;
}
public async Task AddFamilyDataAsync(Family family)
{
await this._container.CreateItemAsync<Family>(family, new PartitionKey(family.Address.ZipCode));
}
public async Task UpdateFamilyDataAsync(Family family)
{
await this._container.ReplaceItemAsync<Family>(family, family.Id, new PartitionKey(family.Address.ZipCode));
}
public async Task DeleteFamilyDataAsync(string id, string zipCode)
{
await this._container.DeleteItemAsync<Family>(id, new PartitionKey(zipCode));
}
}
If you are searching for something with the partition key and id then that can only return a single item as id must be unique within a pk. So GetFamilyDataAsync() can be implemented using ReadItemAsync() rather than a query. PS: Not sure why you would ever do this, allowSynchronousQueryExecution: true
GetFamilyDataFromId() also can be implemented using ReadItemAsync(). Even if this did require a query, it doesn't make sense to implement to return a stream, then deserialize into an object. Just use the variant that can deserialize directly into a POCO.
Can't say if this is completely optimal for everything, but you can go here to find more things as a place to start, https://learn.microsoft.com/en-us/azure/cosmos-db/sql/best-practice-dotnet
The other thing to call out is zip code as a partition key. This might work at small scale, but it could run into issues at larger scale depending on how many families and how many records for each family are stored within a single partition. Max size for a single partition key value is 20 GB. Fine for small workloads but possibly an issue at larger scale.

Why are my swagger docs showing 'additionalProperties = false' for my custom schema filter?

I have this SchemaFilter in my swagger config
public class SmartEnumSchemaFilter : ISchemaFilter
{
public void Apply(OpenApiSchema schema, SchemaFilterContext context)
{
if (!TryGetSmartEnumValues(context.Type, out var values))
{
return;
}
var openApiInts = new OpenApiArray();
openApiInts.AddRange(values.Select(x => new OpenApiInteger(x.Value)));
schema.Type = "integer";
schema.Enum = openApiInts;
schema.Properties = null;
schema.Description = string.Join(", ", values.Select(v => $"{v.Value}: {v.Name}"));
}
}
It is working well, but for some reason this "additional properties" always appears in the docs:
But only for this particular schema - all the other schemas don't have it. Is there some way of removing it?
I tried setting both
schema.AdditionalProperties = null;
schema.AdditionalPropertiesAllowed = false;
but it made no difference

error CS0103: The name 'ModelHelper' does not exist in the current context

I am trying to implement Object detection on HoloLens2 using Microsoft Custom Vision.
I have been following the tutorial (https://learn.microsoft.com/en-us/archive/blogs/appconsult/23535)
but faced with this error..
error CS0103: The name 'ModelHelper' does not exist in the current context
the codes are as below.
what is this ModelHelper for?
and is there anything that i have to add to use it?
using System;
using System.Linq;
using System.Threading.Tasks;
#if UNITY_WSA && !UNITY_EDITOR
using Windows.Media.Capture;
using Windows.Media.Capture.Frames;
using Windows.Media.MediaProperties;
public class ScanEngine
{
public TimeSpan PredictionFrequency = TimeSpan.FromMilliseconds(400);
private MediaCapture CameraCapture;
private MediaFrameReader CameraFrameReader;
private Int64 FramesCaptured;
IUnityScanScene UnityApp;
public ScanEngine()
{
}
public async Task Inititalize(IUnityScanScene unityApp)
{
UnityApp = unityApp;
await InitializeCameraCapture();
await InitializeCameraFrameReader();
}
private async Task InitializeCameraCapture()
{
CameraCapture = new MediaCapture();
MediaCaptureInitializationSettings settings = new
MediaCaptureInitializationSettings();
settings.StreamingCaptureMode = StreamingCaptureMode.Video;
await CameraCapture.InitializeAsync(settings);
}
private async Task InitializeCameraFrameReader()
{
var frameSourceGroups = await MediaFrameSourceGroup.FindAllAsync();
MediaFrameSourceGroup selectedGroup = null;
MediaFrameSourceInfo colorSourceInfo = null;
foreach (var sourceGroup in frameSourceGroups)
{
foreach (var sourceInfo in sourceGroup.SourceInfos)
{
if (sourceInfo.MediaStreamType == MediaStreamType.VideoPreview
&& sourceInfo.SourceKind == MediaFrameSourceKind.Color)
{
colorSourceInfo = sourceInfo;
break;
}
}
if (colorSourceInfo != null)
{
selectedGroup = sourceGroup;
break;
}
}
var colorFrameSource = CameraCapture.FrameSources[colorSourceInfo.Id];
var preferredFormat = colorFrameSource.SupportedFormats.Where(format =>
{
return format.Subtype == MediaEncodingSubtypes.Argb32;
}).FirstOrDefault();
CameraFrameReader = await CameraCapture.CreateFrameReaderAsync(colorFrameSource);
await CameraFrameReader.StartAsync();
}
public void StartPullCameraFrames()
{
Task.Run(async () =>
{
for (; ; ) // Forever = While the app runs
{
FramesCaptured++;
await Task.Delay(PredictionFrequency);
using (var frameReference = CameraFrameReader.TryAcquireLatestFrame())
using (var videoFrame = frameReference?.VideoMediaFrame?.GetVideoFrame())
{
if (videoFrame == null)
{
continue; //ignoring frame
}
if (videoFrame.Direct3DSurface == null)
{
videoFrame.Dispose();
continue; //ignoring frame
}
try
{
await
ModelHelper.EvaluateVideoFrameAsync(videoFrame).ConfigureAwait(false);
}
catch (Exception ex)
{
System.Diagnostics.Debug.WriteLine(ex.Message);
}
finally
{
}
}
}
});
}
}
#endif

How to use Scroll while passing raw json Query to ElasticSearch using NEST

var query = #"
{
""query"": {
""match_all"": { }
}
}";
Func<SearchRequestParameters, SearchRequestParameters> requestParameters = a =>
a.SearchType(SearchType.Scan).Scroll(TimeSpan.FromSeconds(60));
var searchResult = await client.LowLevel.SearchAsync<SearchResponse<T>>(indexName, mappingName, query , requestParameters)
if (searchResult.Body.IsValid)
{
var scrollNo = 0;
var results = await client.ScrollAsync<T>("10s", searchResult.Body.ScrollId);
while (results.Documents.Any())
{
documents.AddRange(results.Documents);
scrollNo++;
results = await client.ScrollAsync<T>("10s", results.ScrollId);
return new Customresponse<T>
{
Documents = documents,
total = result.Body.Total
};
}
Would like to pull all data using scroll while passing raw json query. but scroll is not working properly while passing json raw query. Can anyone help on this ?.
Your example is nearly there but not quite; you're missing a closing brace for the while loop to collect all documents before returning the custom response.
Here's an example I just ran on the Stackoverflow data set, to return all questions tagged with nest
private IElasticClient _client;
void Main()
{
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var defaultIndex = "default-index";
var connectionSettings = new ConnectionSettings(pool);
_client = new ElasticClient(connectionSettings);
var query = #"
{
""query"": {
""term"": {
""tags"": {
""value"": ""nest""
}
}
}
}";
var result = RunScrollAsync(query).Result.Dump();
}
private async Task<Customresponse<Question>> RunScrollAsync(string query)
{
var scrollTime = "10s";
// omit the .SearchType(Scan) which is deprecated. Not
// specifying means the first response contains the first set
// of documents
var esResponse = await _client.LowLevel.SearchAsync<SearchResponse<Question>>(
"posts",
"question",
query, r => r.Scroll(TimeSpan.FromSeconds(10))).ConfigureAwait(false);
if (esResponse.Body.IsValid && esResponse.Body.Documents.Any())
{
// assume you have less than 2,147,483,647 documents to return?
var documents = new List<Question>((int)esResponse.Body.Total);
documents.AddRange(esResponse.Body.Documents);
var scrollNo = 0;
var response = await _client.ScrollAsync<Question>(scrollTime, esResponse.Body.ScrollId).ConfigureAwait(false);;
// keep scrolling until no more documents are returned
while (response.Documents.Any())
{
documents.AddRange(response.Documents);
scrollNo++;
response = await _client.ScrollAsync<Question>(scrollTime, response.ScrollId).ConfigureAwait(false);;
}
return new Customresponse<Question>
{
Documents = documents,
total = response.Total
};
}
// return an empty result.
// Or throw an exception, or log - whatever you need to do
return new Customresponse<Question>
{
Documents = Enumerable.Empty<Question>(),
total = 0
};
}
public class Customresponse<T>
{
public IEnumerable<T> Documents { get; set; }
public long total { get; set; }
}
This returns all 342 questions, with a total of 342 (Data set is from June 2016).

Restful WebApi V2 Put Issue

I cannot understand why my WebApi returns a success (200) when I Post, but doesn't return anything when I Put. Both successfully update the database.
I have attached my code for both methods.
Thanks
Paul
// PUT: api/items/5
[Route("api/items/{id:int}")]
[ResponseType(typeof(void))]
public async Task<IHttpActionResult> PutItem([FromBody] Item[] item, string updateitems)
{
if (!ModelState.IsValid)
{
return BadRequest(ModelState);
}
foreach (Item i in item)
{
if (updateitems.IndexOf(i.ItemId.ToString()) > 0)
{
db.Entry(i).State = EntityState.Modified;
}
}
await db.SaveChangesAsync();
var iStatus = StatusCode(HttpStatusCode.OK);
return iStatus;
}
// POST: api/items
[Route("api/items")]
[ResponseType(typeof(Item))]
public async Task<IHttpActionResult> PostItem([FromBody] Item[] item)
{
int itemCount = 0;
string edititems = "";
IHttpActionResult iStatus;
if (!ModelState.IsValid)
{
return BadRequest(ModelState);
}
foreach (Item i in item)
{
if (!ItemExists(i.SurveyId, i.LineId))
{
db.Items.Add(i);
}
else
{
var id = FindItemId(i.SurveyId, i.LineId);
edititems = edititems + ',' + id.ToString();
i.ItemId = id;
}
itemCount ++;
}
await db.SaveChangesAsync();
if (edititems.Length > 0)
{
IHttpActionResult result = PutItem(item, edititems).Result;
}
iStatus = StatusCode(HttpStatusCode.OK);
return iStatus;
}
your code looks fine, the only difference that I see is in the ResponseType attribute of the Put, you have: [ResponseType(typeof(void))]
Change to: [ResponseType(typeof(item))]