I'm able to do affinity computer-data collocation with apache ignite. In the following two examples, it works as expected.
// Works on all nodes
IgniteUtil.getIgnite().compute().broadcast(() -> {
System.out.println("Should happen on all nodes");
cache.get(key).forEach(x -> {
System.out.println(x);
});
});
// Works on just the one node
IgniteUtil.getIgnite().compute().affinityRun(IgniteUtil.CACHE_NAME, key , () -> {
System.out.println("Should only happen on one node");
cache.get(key).forEach(x -> System.out.println(x));
});
However, I want to run a lambda against all the nodes data. So for example, say I had cached For every person, all of their orders from Amazon. I want to know what the total dollar amount is for everyone's orders.
I'm probably just missing an example, but according to the docs I don't see how to do this. In the examples I've seen I have to specify the keys I want to compute with. In this example, I just want to be able to do some lambda on all of the nodes, with each node only operating on its own share of the data.
I've tried doing this
IgniteUtil.getIgnite().compute().affinityRun(IgniteUtil.CACHE_NAME, key , () -> {
System.out.println("Should only happen once per node");
List<Integer> count = new ArrayList<Integer>();
System.out.println("Size: " + Sets.newHashSet(cache.iterator()).size());
cache.iterator().forEachRemaining(x -> {count.add(count.size());});
System.out.println("Calculated Size: " + count.size());
System.out.println("Values: " );
cache.get(key).forEach(x -> System.out.print(x));
System.out.println();
});
and it does only execute on the node that has the key, however, the cache size is the full cache size, not just the values that are local.
Any suggestions?
You can broadcast a closure as in your first example and use IgniteCache.localEntries() method to iterate through the local data.
Related
Im bulding a game server where i'm holding a list of connected users and at the same time a list of battles, those 2 lists are seperate.
I'm holding the battles data in Redis db, with battle id as the key,
could I use the connected user list in the same Redis db and get a list of the connected users in a elegant way while avoining battles data ?
I couldn't find a solution for that and currently holding the connected users in JS dictionary in my nodeJs server
UPDATE:
to clarify queries,
connected user object is -
connectedUser = {
nickname: string, (key)
socketId: string,
status: string,
rating: number
}
i would like to query the data with getUser(nickname) which will fetch a specific user and getAllUsers() to fetch all connected users.
as for match it includes all match data and stats
match = {
id: string, (key)
nicknameOne: string,
nicknameTwo: string,
...
}
on matches I query with id and believed it works well with hash table and hset and hget complexity wize.
as for queries on connected user, I wonder if using a simple js dictionary might provide with faster query results.
the js dictionary looks like that -
connectedUsers = {}
adding user -
user = {
key: nickname,
value: {
socketId: string,
status: string,
rating: number
}
}
while getting a specific user like that -
connectedUsers[nickname]
and iterating all this way -
for(let key in connectedUsers) {
//value - connectedUsers[key];
}
Nothing will beat the JS dictionary in terms of performance. Even with something as fast as Redis you're adding the overhead of interprocess communication (if hosted on the same machine) or network latencies.
If you want to use Redis for persistence, one way to store connected users would be to have a hash where keys are nicknames and values are user objects. You can do HGET users-key userId to get 1 user and HGETALL users-key to fetch all users.
I am working on an application that fetches data from a database and I would like to show them as graph.
I managed the "tap" event on a node by showing their neighbors (nodes and connection links).
The problem is that, every time I want to show the neighbors, all the graph is re-rendered and if some nodes were moved before, they lose their previous position.
Is there a way to add only the neighbors without affecting the position of the node already present in the layout?
Important: the constraint is that all the nodes should be "movable": the number of nodes in the graph can, easily, increase and I would like to have the availability to move/organize them without losing the result when I add new ones (by clicking on a node)
I am using cola-layout in my project.
Here the way I managed to add neighbors:
function addNeighbour(node, link) {
cy.startBatch();
addNode(link.otherNode.type, link.otherNode.name, link.otherNode.properties);
cy.add([
{
group: 'edges',
data:
{
id: node + ":" + link.type + ":" + link.otherNode.type + ":" + link.otherNode.name,
source: source,
target: target,
type: link.type,
properties: linkproperties
}
}
]);
refreshLayout()
cy.endBatch();
}
}
var layoutOpts = {
name: 'cola',
refresh: 2,
edgeLength: 200,
fit: false
}
function refreshLayout() {
layout.stop();
layout = cy.elements().makeLayout(layoutOpts);
layout.run();
}
Thanks in advance
(1) You can lock a node to make its position immutable, via nodes.lock().
(2) You can run a layout on a subset of the graph to exclude certain elements, via eles.layout().
Either of these strategies can be used in general, or they can be used in tandem.
For your case, it sounds like you should use (1).
Lock the existing nodes.
Add the new nodes.
Run Cola on the entire graph.
When Cola is done, free the locked nodes.
Note, however, that this won't always give a good result. You could over-constrain the system. If you want a good layout result, it's probably best to just run the layout on everything without locking, as Stephan T. suggested.
I am struggling with using observables and pipes where I want to add one item from one observable to another observable containing a list of items of the same type.
I have type X. And of the type X I have an observable array:
readonly arrayOfx$: Observable<X[]>;
I also have an observable of only the the type X:
private readonly _x$: Observable<UpdateOfX>;
interface UpdateOfX {
x: X,
updateState: "Add" | "Modified" | "Removed"
}
All this code is in a Service Class where the service should only expose the array of X. The data in the array I want to show in my html with async piping and this part of the functionality works. The host and the client are connected with the signalR technique and onConnected, an array of items of type X is retrieved. But as the application runs, in the backend new items of type X can be created, existing items can be changed or can be removed and when this occures, only this item will be send via the signalR connection and the modification state.
In the front end, this item must be added to the already retrieved array of items of type X. In the service, the pipe technique is used and my question is, how do I add the single item I get in a later moment to the list of items I retrieved earlier?
constructor() {
this.arrayOfx$ = this._someSignalRHelperService.retrieveMultipleItems$.pipe(
tap((xArray: X[]) => console.log(xArray)),
//can I somehow get the a later created x from the server here...
);
this._x$ = this._someSignalRHelperService.retrieveOneItem$.pipe(
tap((updateOfX: UpdateOfX) => console.log(updateOfX)),
map((updateOfX: UpdateOfX) => {
//process the updateState
//... or must I do something here to get x into x[]?
})
);
}
Since SignalR is used, the backend is in control when the client receives a new item of type X when there is one created.
You can use combineLatest(), and do whatever manipulation you want as soon as your receive the two emissions:
constructor() {
this.combinedOfX$ = combineLatest(
this._someSignalRHelperService.retrieveMultipleItems$,
this._someSignalRHelperService.retrieveOneItem$
).pipe(
map(([singleOfX, multipleOfX]) => {
// do your adding or mapping and whatever here.
})
)
}
Not sure if that is by design but the problem with having the frontend (client) to deal with the data is not so ideal. Your single source of truth is now based on your client, which different machines have different processing speed and may cause nuances and inconsistent displays. Also, the code will be messy in a sense that you will now need to check through the entire arrayOfX every time a singleOfX gets updated - you need to check if it exists in the current list, if yes, edit/delete; else, append to the list. What if the user refreshes his browser accidentally? You lost all of your processing.
Since you are already using SignalR, it would be more advisable that you let the server handle all the data, and let the server be single source of truth. Then you will just need to subscribe to one hub and listen to the changes of the arrayOfX; and pretty much don't care of the single updates.
When the key is a composite of id1, id2 in a GemFire Region and the Region is partitioned with id1, what is the best way of getting all the rows whose key matched id1.
Couple of options that we are thinking of:
Create another index on id1. If we do that, we are wondering if it goes against all Partition Regions?
Write data aware Function and Filter by (id1, null) to target specific Partition Region. Use index in local Region by using QueryService?
Can you please let me know if there is any other way to achieve the query by partial key.
Well, it could be implemented (optimally) by using a combination of #1 and #2 in your "options" above (depending on whether your application domain object also stored/referenced the key, which would be the case if you were using SD[G] Repositories.
This might be best explained with the docs and an example, particularly using the PartitionResolver interface Javadoc.
Say your "composite" Key was implemented as follows:
class CompositeKey implements PartitionResolver {
private final Object idOne;
private final Object idTwo;
CompositeKey(Object idOne, Object idTwo) {
// argument validation as necessary
this.idOne = idOne;
this.idTwo = idTwo;
}
public String getName() {
return "MyCompositeKeyPartitionResolver";
}
public Object getRoutingObject() {
return idOne;
}
}
Then, you could invoke a Function that queries the results you desire by using...
Execution execution = FunctionService.onRegion("PartitionRegionName");
Optionally, you could use the returned Execution to filter on just the (complex) Keys you wanted to query (further qualify) when invoking the Function...
ComplexKey filter = { .. };
execution.withFilter(Arrays.stream(filter).collect(Collectors.toSet()));
Of course, this is problematic if you do not know your keys in advance.
Then you might prefer to use the ComplexKey to identify your application domain object, which is necessary when using SD[G]'s Repository abstraction/extension:
#Region("MyPartitionRegion")
class ApplicationDomainObject {
#Id
CompositeKey identifier;
...
}
And then, you can code your Function to operate on the "local data set" of the Partition Region. That is, when a data node in the cluster hosts the same Partition Region (PR), then it will only operate on the data set in the "bucket" for that PR, which is accomplished by doing the following:
class QueryPartitionRegionFunction implements Function {
public void execute(FunctionContext<Object> functionContext) {
RegionFunctionContext regionFunctionContext =
(RegionFunctionContext) functionContext;
Region<ComplexKey, ApplicationDomainObject> localDataSet =
PartitionRegionHelper.getLocalDataForContext(regionFunctionContext);
SelectResults<?> resultSet =
localDataSet.query(String.format("identifier.idTwo = %s",
regionFunctionContext.getArguments);
// process result set and use ResultSender to send results
}
}
Of course, all of this is much easier to do using SDG's Function annotation support (i.e. implementing and invoking your Function anyway).
Note that, when you invoke the Function, onRegion using the GemFire's FunctionService, or more conveniently with SDG's annotation support for Function Execution, like so:
#OnRegion("MyPartitionRegion")
interface MyPartitionRegionFunctions {
#FunctionId("QueryPartitionRegion")
<return-type> queryPartitionRegion(..);
}
Then..
Object resultSet = myPartitionRegionFunctions.queryPartitionRegion(..);
Then, the FunctionContext will be a RegionFunctionContext (because you executed the Function on the PR, which executes on all nodes in the cluster hosting the PR).
Additionally, you use the PartitionRegionHelper.getLocalDataForContext(:RegionFunctionContext) to get the local data set of the PR (i.e. the bucket, or just the shard of data in the entire PR (across all nodes) hosted by that node, which would be based your "custom" PartitionResolver).
You can then query to further qualify, or filter the data of interests. You can see that I queried (or further qualified) by idTwo, which was not part of the PartitionResolver implementation. Additionally, this would only be required in the (OQL) query predicate if you did not specify Keys in your Filter with the Execution (since, I think, that would take the entire "Key" (idOne & idTwo) into account, based on our properly implemented Object.equals() method of your ComplexKey class).
But, if you did not know the keys in advance and/or (especially if) you are using SD[G]'s Repositories, then the ComplexKey would be part of your application domain abject, which you could then Index, and query on (as shown above: identifier.idTwo = ?).
Hope this helps!
NOTE: I have not test any of this, but hopefully it will point you in the right direction and/or give you further ideas.
I want to authorize a storage with VMs. For that I need to have all the VM ID's for a storage and those I get using the following call:
https://[username]:[apikey]#api.softlayer.com/rest/v3/SoftLayer_Network_Storage_Iscsi/9653497/getAllowableVirtualGuests?objectMask=mask[id,fullyQualifiedDomainName]
This gives me all the VM ID's corresponding to 9653497 (storage/order ID). However, I need to have all those storage ID's (like 9653497) which are not assigned to any of the VM's ID. I am using below call to get all storage ID:
https://[username]:[apikey]#api.softlayer.com/rest/v3/SoftLayer_Account/getNetworkStorage?objectMask=mask[id,username,nasType,storageType, billingItem[description,location[id,longName]]]&objectFilter={"networkStorage":{"nasType":{"operation":"ISCSI"},"billingItem":{"description":{"operation":"Endurance Storage"}}}}
the data that you are using for the filter probably are wrong, try to call the get object method GET /SoftLayer_Network_Storage/9653497/getObject?objectMask=mask[nasType,billingItem[description]] and see if the values of the request are the same as of your objectFilter
The filter in your request, gets Block Storage("nasType":{"operation":"ISCSI"}), maybe you need the File Storage. We can remove it to get more "Endurance" items (Block and File).
Please try the following removing some filters:
https://[username]:[apikey]#api.softlayer.com/rest/v3/SoftLayer_Account/getNetworkStorage?objectMask=mask[id,username,nasType,storageType, billingItem[description,location[id,longName]]]&objectFilter={ "networkStorage": { "billingItem": { "description": { "operation": "Endurance Storage" } } } }
Method: GET
if we don't want to get only Endurance, we can remove that filter too.
But when trying to add some properties using objectMasks to SoftLayer_Account::getNetworkStorage like allowableVirtualGuests, that property is not present in SoftLayer_Network_Storage.
For that reason the unique way to get “getAllowableVirtualGuests” is using SoftLayer_Network_Storage::getAllowableVirtualGuests