When the key is a composite of id1, id2 in a GemFire Region and the Region is partitioned with id1, what is the best way of getting all the rows whose key matched id1.
Couple of options that we are thinking of:
Create another index on id1. If we do that, we are wondering if it goes against all Partition Regions?
Write data aware Function and Filter by (id1, null) to target specific Partition Region. Use index in local Region by using QueryService?
Can you please let me know if there is any other way to achieve the query by partial key.
Well, it could be implemented (optimally) by using a combination of #1 and #2 in your "options" above (depending on whether your application domain object also stored/referenced the key, which would be the case if you were using SD[G] Repositories.
This might be best explained with the docs and an example, particularly using the PartitionResolver interface Javadoc.
Say your "composite" Key was implemented as follows:
class CompositeKey implements PartitionResolver {
private final Object idOne;
private final Object idTwo;
CompositeKey(Object idOne, Object idTwo) {
// argument validation as necessary
this.idOne = idOne;
this.idTwo = idTwo;
}
public String getName() {
return "MyCompositeKeyPartitionResolver";
}
public Object getRoutingObject() {
return idOne;
}
}
Then, you could invoke a Function that queries the results you desire by using...
Execution execution = FunctionService.onRegion("PartitionRegionName");
Optionally, you could use the returned Execution to filter on just the (complex) Keys you wanted to query (further qualify) when invoking the Function...
ComplexKey filter = { .. };
execution.withFilter(Arrays.stream(filter).collect(Collectors.toSet()));
Of course, this is problematic if you do not know your keys in advance.
Then you might prefer to use the ComplexKey to identify your application domain object, which is necessary when using SD[G]'s Repository abstraction/extension:
#Region("MyPartitionRegion")
class ApplicationDomainObject {
#Id
CompositeKey identifier;
...
}
And then, you can code your Function to operate on the "local data set" of the Partition Region. That is, when a data node in the cluster hosts the same Partition Region (PR), then it will only operate on the data set in the "bucket" for that PR, which is accomplished by doing the following:
class QueryPartitionRegionFunction implements Function {
public void execute(FunctionContext<Object> functionContext) {
RegionFunctionContext regionFunctionContext =
(RegionFunctionContext) functionContext;
Region<ComplexKey, ApplicationDomainObject> localDataSet =
PartitionRegionHelper.getLocalDataForContext(regionFunctionContext);
SelectResults<?> resultSet =
localDataSet.query(String.format("identifier.idTwo = %s",
regionFunctionContext.getArguments);
// process result set and use ResultSender to send results
}
}
Of course, all of this is much easier to do using SDG's Function annotation support (i.e. implementing and invoking your Function anyway).
Note that, when you invoke the Function, onRegion using the GemFire's FunctionService, or more conveniently with SDG's annotation support for Function Execution, like so:
#OnRegion("MyPartitionRegion")
interface MyPartitionRegionFunctions {
#FunctionId("QueryPartitionRegion")
<return-type> queryPartitionRegion(..);
}
Then..
Object resultSet = myPartitionRegionFunctions.queryPartitionRegion(..);
Then, the FunctionContext will be a RegionFunctionContext (because you executed the Function on the PR, which executes on all nodes in the cluster hosting the PR).
Additionally, you use the PartitionRegionHelper.getLocalDataForContext(:RegionFunctionContext) to get the local data set of the PR (i.e. the bucket, or just the shard of data in the entire PR (across all nodes) hosted by that node, which would be based your "custom" PartitionResolver).
You can then query to further qualify, or filter the data of interests. You can see that I queried (or further qualified) by idTwo, which was not part of the PartitionResolver implementation. Additionally, this would only be required in the (OQL) query predicate if you did not specify Keys in your Filter with the Execution (since, I think, that would take the entire "Key" (idOne & idTwo) into account, based on our properly implemented Object.equals() method of your ComplexKey class).
But, if you did not know the keys in advance and/or (especially if) you are using SD[G]'s Repositories, then the ComplexKey would be part of your application domain abject, which you could then Index, and query on (as shown above: identifier.idTwo = ?).
Hope this helps!
NOTE: I have not test any of this, but hopefully it will point you in the right direction and/or give you further ideas.
Related
I am learning a course topic reflection and metadata from nest documentation. They used #setmetadata('roles') but I don't know metadata come from and when they are used?
I don't know metadata come from
First lets explain what metadata generally means.
Metadata in general means data about data. Its a description of the data in more simpler terms (for e.g data about an image). Taking an example from here.
They used #setmetadata('roles').
Nest provides the ability to attach custom data to route handlers through #SetMetadata. Its a way to declaratively define and store data about your controller(endpoint).
#SetMetadata stores the key value pairs. For example,
SetMetadata('IS_PUBLIC_KEY', true)
findAll(#Query() paginationQuery: PaginationQueryDto) {
return this.testService.findAll(paginationQuery);
}
Here I am setting a key IS_PUBLIC_KEY with a value set to true.
In this scenario you are defining a key named role (most probably and it seems it may be missing a value) which will define what certain types or role can access this controller.
When they are used?
You can use it when you want to define the Guards. For instance, I am using the above findAll controller as a public api. In my guard implementation, I check and see if the value of IsPublic is true then allow any consumer to consume the API.
canActivate(
context: ExecutionContext,
): boolean | Promise<boolean> | Observable<boolean> {
const isPublic = this.reflector.get('IS_PUBLIC_KEY', context.getHandler());
if (isPublic) {
return true;
}
}
Hope this answers your question.
https://docs.nestjs.com/fundamentals/execution-context#reflection-and-metadata:
The #SetMetadata() decorator is imported from the #nestjs/common package.
I need to query a property of Saga Data class to get a list. It is stored on SqlPersistance table [Data] column as a serialized object.
Think about a scenario that my SagaData has a property called UserName, so I want to query every saga related to that user.
In a sloppy way, I can query the column content, get the list and can create Saga objects out of the content, by querying like:
SELECT [Id]
,[Correlation_TaskId]
,[Metadata]
,[Data]
,[PersistenceVersion]
,[SagaTypeVersion]
,[Concurrency]
FROM [myWonderfulDb].[dbo].[MyWonderfulPeristanceTable]
where JSON_VALUE(Data,'$.Username') = 'arthur'
but I am looking for an elegant way to do it by possibly using NserviceBus API's.
There is a SagaFinder implementation described in ParticularSoftware documentation (link: https://docs.particular.net/persistence/sql/saga-finder) but this returns only one object which does not perfectly fit into my scenario.
Here how it is implemented in the documentation:
class SqlServerSagaFinder :
IFindSagas<MySagaData>.Using<MyMessage>
{
public Task<MySagaData> FindBy(MyMessage message, SynchronizedStorageSession session, ReadOnlyContextBag context)
{
return session.GetSagaData<MySagaData>(
context: context,
whereClause: "JSON_VALUE(Data,'$.PropertyPathInJson') = #propertyValue",
appendParameters: (builder, append) =>
{
var parameter = builder();
parameter.ParameterName = "propertyValue";
parameter.Value = message.PropertyValue;
append(parameter);
});
}
}
Any ideas appreciated. Thank you!
We have guidance about querying saga state available at
https://docs.particular.net/nservicebus/sagas/#querying-saga-data
In short, you can query the saga data, there is no out of the box way provided by NServiceBus because we recommend using a different approach instead:
the saga to publish events containing the required data and have handlers that process these events and store the data in one or more read model(s) for querying purposes.
The major points why we don't recommend it is in my viewpoint these two
By exposing the data outside of the safeguards of the business logic in the saga the risk is that the data is not treated as
read-only. Eventually, a component tries to bypass the saga and
directly modify the data.
Querying the data might require additional indexes, resources etc. which need to be managed by the component issuing the query.
Those additional resources can influence saga performance.
The only purpose of the SagaFinder is to find a single instance of a certain saga that correlates with the incoming message. I'm not aware of any framework functionalities to query saga (data) instances.
I'm trying to use RepoDb to query the contents of a table (in an existing Sql Server database), but all my attempts result in an InvalidOperationException (There are no 'contructor parameter' and/or 'property member' bindings found between the resultset of the data reader and the type 'MyType').
The query I'm using looks like the following:
public Task<ICollection<MyType>> GetAllAsync()
{
var result = new List<MyType>();
using (var db = new SqlConnection(connectionString).EnsureOpen())
{
result = (await db.ExecuteQueryAsync<MyType>("select * from mytype")).ToList();
}
return result;
}
I'm trying to run this via a unit test, similar to the following:
[Test]
public async Task MyTypeFetcher_returns_all()
{
SqlServerBootstrap.Initialize();
var sut = new MyTypeFetcher("connection string");
var actual = await sut.GetAllAsync();
Assert.IsNotNull(actual);
}
The Entity I'm trying to map to matches the database table (i.e. class name and table name are the same, property names and table column names also match).
I've also tried:
putting annotations on the class I am trying to map to (both at the class level and the property level)
using the ClassMapper to map the class to the db table
using the FluentMapper to map the entire class (i.e. entity-table, all columns, identity, primary)
putting all mappings into a static class which holds all mapping and configuration and calling that in the test
providing mapping information directly in the test via both ClassMapper and FluentMapper
From the error message it seems like RepoDb cannot find the mappings I'm providing. Unfortunately I have no idea how to go about fixing this. I've been through the documentation and the sample tutorials, but I haven't been able to find anything of use. Most of them don't seem to need any mapping configuration (similar to what you would expect when using Dapper). What am I missing, and how can I fix this?
I am using TYPO3 8. In my extension I have a database table "company" in which I store for each company the total number of places (number_places) and the number of occupied places (occupied_places).
Now I want to limit the search to companies which have available places left.
In MySQL it would be like this:
SELECT * FROM company WHERE number_places > occupied_places;
How can I create this query in the extbase repository?
I tried to introduce the virtual property placesLeft in my model but it did not work.
I don't want to use a raw SQL statement as mentioned below, because I already have implemented a filter which uses plenty of different constraints.
Extbase query to compare two fields in same table
You can do it like this in your repository class, please note the comments inside the code:
class CompanyRepository extends \TYPO3\CMS\Extbase\Persistence\Repository
{
public function findWithAvailablePlaces(bool $returnRawQueryResult = false)
{
// Create a QueryBuilder instance
$queryBuilder = $this->objectManager->get(\TYPO3\CMS\Core\Database\ConnectionPool::class)
->getConnectionForTable('company')->createQueryBuilder();
// Create the query
$queryBuilder
->select('*')
->from('company')
->where(
// Note: this string concatenation is needed, because TYPO3's
// QueryBuilder always escapes the value in the ExpressionBuilder's
// methods (eq(), lt(), gt(), ...) and thus render it impossible to
// compare against an identifier.
$queryBuilder->quoteIdentifier('number_places')
. \TYPO3\CMS\Core\Database\Query\Expression\ExpressionBuilder::GT
. $queryBuilder->quoteIdentifier('occupied_places')
);
// Execute the query
$result = $queryBuilder->execute()->fetchAll();
// Note: this switch is not needed in fact. I just put it here, if you
// like to get the Company model objects instead of an array.
if ($returnRawQueryResult) {
$dataMapper = $this->objectManager->get(\TYPO3\CMS\Extbase\Persistence\Generic\Mapper\DataMapper::class);
return $dataMapper->map($this->objectType, $result);
}
return $result;
}
}
Notes:
If you have lots of records to deal with, I would - for performance reasons - not use the data mapping feature and work with arrays.
If you want to use the fluid pagination widget, be sure you don't and build your own pagination. Because of the way this works (extbase-internally), you'd get a huge system load overhead when the table grows. Better add the support for limited db queries to the repository method, for example:
class CompanyRepository extends \TYPO3\CMS\Extbase\Persistence\Repository
{
public function findWithAvailablePlaces(
int $limit = 10,
int $offset = 0,
bool $returnRawQueryResult = false
) {
// ...
$queryBuilder
->setMaxResults($limit)
->setFirstResult($offset);
$result = $queryBuilder->execute()->fetchAll();
// ...
}
}
I think you cant do this using the default Extbase Query methods like equals() and so on. You may use the function $query->statement() for your specific queries like this.
You also can use the QueryBuilder since TYPO3 8 which has functions to compare fields to each other:
https://docs.typo3.org/typo3cms/CoreApiReference/latest/ApiOverview/Database/QueryBuilder/Index.html#quoteidentifier-and-quoteidentifiers
It's fine to use this QueryBuilder inside Extbase repositories. After this you can use the DataMapper to map the query results to Extbase models.
In case of using "statement()" be aware of escaping every value which may cause any kind of SQL injections.
Based on the current architecture of TYPO3, the data structure is such that comparing of two tables or, mixing results from two tables ought to be done from within the controller, by injecting the two repositories. Optionally, you can construct a Domain Service that can work on the data from the two repositories from within the action itself, in the case of a routine. The service will also have to be injected.
Note:
If you have a foreign relation defined in your table configuration, the results of that foreign relation will show in your defined table repository. So, there's that too.
All,
I have a requirement to hide my EF implementation behind a Repository. My simple question: Is there a way to execute a 'find' across both a DbSet AND the DbSet.Local without having to deal with them both.
For example - I have standard repository implementation with Add/Update/Remove/FindById. I break the generic pattern by adding a FindByName method (for demo purposes only :). This gives me the following code:
Client App:
ProductCategoryRepository categoryRepository = new ProductCategoryRepository();
categoryRepository.Add(new ProductCategory { Name = "N" });
var category1 = categoryRepository.FindByName("N");
Implementation
public ProductCategory FindByName(string s)
{
// Assume name is unique for demo
return _legoContext.Categories.Where(c => c.Name == s).SingleOrDefault();
}
In this example, category1 is null.
However, if I implement the FindByName method as:
public ProductCategory FindByName(string s)
{
var t = _legoContext.Categories.Local.Where(c => c.Name == s).SingleOrDefault();
if (t == null)
{
t = _legoContext.Categories.Where(c => c.Name == s).SingleOrDefault();
}
return t;
}
In this case, I get what I expect when querying against both a new entry and one that is only in the database. But this presents a few issues that I am confused over:
1) I would assume (as a user of the repository) that cat2 below is not found. But it is found, and the great part is that cat2.Name is "Goober".
ProductCategoryRepository categoryRepository = new ProductCategoryRepository();
var cat = categoryRepository.FindByName("Technic");
cat.Name = "Goober";
var cat2 = categoryRepository.FindByName("Technic");
2) I would like to return a generic IQueryable from my repository.
It just seems like a lot of work to wrap the calls to the DbSet in a repository. Typically, this means that I've screwed something up. I'd appreciate any insight.
With older versions of EF you had very complicated situations that could arise quite fast due to the required references. In this version I would recomend not exposing IQueryable but ICollections or ILists. This will contain EF in your repository and create a good seperation.
Edit: furthermore, by sending back ICollection IEnumerable or IList you are restraining and controlling the queries being sent to the database. This will also allow you to fine tune and maintain the system with greater ease. By exposing IQueriable, you are exposing yourself to side affects which occur when people add more to the query, .Take() or .Where ... .SelectMany, EF will see these additions and will generate sql to reflect these uncontrolled queries. Not confining the queries can result in queries getting executed from the UI and is more complicated tests and maintenance issues in the long run.
since the point of the repository pattern is to be able to swap them out at will. the details of DbSets should be completly hidden.
I think that you're on a good path. The only thing I probaly ask my self is :
Is the context long lived? if not then do not worry about querying Local. An object that has been Inserted / Deleted should only be accessible once it has been comitted.
if this is a long lived context and you need access to deleted and inserted objects then querying the Local is a good idea, but as you've pointed out, you may run into difficulties at some point.