NoFlo Components documentation mentions groups but doesn't explain what they are used for and how they should be used.
Could somebody explain what role groups play in NoFlo, how groups and nested groups should be used and how it affects Asynchronous components?
The concept of groups is described as "bracket IPs" in the FBP book.
Basically begingroup and endgroup are special packet types that specify that data packets sent between them contain that group as metadata.
Think of it a bit like XML:
<somegroup>
<innergroup>
data
</innergroup>
</somegroup>
In NoFlo this would work with:
#outPorts.out.beginGroup 'somegroup'
#outPorts.out.beginGroup 'innergroup'
#outPorts.out.send "data"
#outPorts.out.endGroup()
#outPorts.out.endGroup()
The receiving inport gets these via the begingroup and endgroup events and can either do something with them or ignore them. Usually a good behavior with components that don't utilize groups but perform some transformations on information packets is to at least pass them onwards.
#inPorts.in.on 'begingroup', (group) =>
#outPorts.out.beginGroup group
#inPorts.in.on 'data', (data) =>
# do something and then send
#inPorts.in.on 'endgroup', =>
#outPorts.out.endGroup()
So, groups can be seen as a way to supply some "metadata" for your packets. For example, NoFlo's when filesystem/ReadFile sends the contents of a file out as a packet, it surrounds it with a group named with the file's path.
Groups can also be super-useful for merging asynchronous flows. For example, webserver/Server generates a unique group identifier for every request it gets. If you do database queries or other async operations before responding to a request, you can use these groups for merging the results back before writing a response.
Related
As per this comment, I'm trying to use numeric channel IDs in my telethon code, so that I don't end up spamming the Telegram API with expensive name lookup calls and getting throttled, but I'm having some difficulty.
e.g. assuming I've already instantiated and connected client:
messages = client.get_messages(numeric_channel_id)
...fails with this error:
ValueError: Could not find the input entity for PeerUser(user_id=[numeric_channel_id]) (PeerUser)
I think there's some cacheing going on, because if I do a get_entity call using the account name first, then the get_messages call works. i.e. something like this:
client.get_entity(channel_name_which_belongs_to_numeric_channel_id)
messages = client.get_messages(numeric_channel_id)
That works just fine, but now I'm doing the expensive get_entity(name) call which is what I'm trying to avoid (because it will result in FloodWaitError problems).
Is there any way I can use the numeric ID of a channel to avoid the expensive get_entity call, in this scenario?
I've also tried forcing the entity type to Channel, like this:
channel = Channel(id=numeric_channel_id, title=None, photo=None, date=None)
messages = client.get_messages(channel)
...but the results are the same, except that the error mentions PeerChannel rather than PeerUser
ID usage is not going to work unless you cached the target as you stated, that's the only way to use the integer id.
you must have met the entity from events or manual requests (say, username fetching).
you should be using client.get_input_entity('username')
it will try to search the local cache first for the saved id + hash that equals the passed username, if found it won't do ResolveUsername (heavy one) and use the local access_hash + id and return you an inputPeer. you pass that to any request you want.
you mustn't use id alone unless you're certain you have met its holder, in other words, id you use has to be something you found out from within the library and within the same session, not something you knew/found out externally.
There is no magical way to fetch something with id you claim you know, if you actually know it, the lib has to create (when the access_hash is present) an InputPeer
As the other answer states, fetching by username will always work but is expensive. However note that such a call will fill the cache so it can later be fetched again much more cheaply by ID.
If you really need a stable reference to some entity and cannot rely on the session cache, and want to avoid usernames, the documentation for Entities vs. Input Entities may be helpful.
What it boils down to is, you can do this:
print(await client.get_input_entity('username'))
...which will show something like:
InputPeerChannel(channel_id=1066197625, access_hash=-6302373944955169144)
...and then, the account that made the get_input_entity call will always be able to use the printed result, without the need for it to be in cache:
from telethon.tl.types import InputPeerChannel
USERNAME = InputPeerChannel(channel_id=1066197625, access_hash=-6302373944955169144)
# ...
await client.send_message(USERNAME, 'Hi') # works without cache
I have a doubt when I'm designing a REST API.
Consider I have a Resource "Customer" with two elements in my server, like this:
[
{
name : "Mary",
description : "An imaginary woman very tall."
},
{
name : "John",
description : "Just a guy."
}
]
And I want to make an endpoint, that will accept a GET request with a query. The query will provide a parameter with a value that will make an algorithm count how many occurrences for this text are there in all of its parameters.
So if we throw this request:
GET {baseURL}/customers?letters=ry
I should get something like
[
{
name : "Mary",
description : "An imaginary woman very tall.",
count : 3
},
{
name : "John",
description : "Just a guy.",
count : 0
}
]
Count parameter can not be included in the resource scheme as will depend on the value provided in the query, so the response objects have to be enriched.
I'm not getting a list of my resource but a modified resource.
Although it keeps the idempotent condition for GET Method, I see it escapes from the REST architecture concept (even the REST beyond CRUD).
Is it still a valid endpoint in a RESTful API? or should I create something like a new resource called "ratedCustomer"?
REST GET mehod: Can return a list of enriched resources?
TL;DR: yes.
Longer answer...
A successful GET request returns a representation of a single resource, identified by the request-target.
The fact that the information used to create the representation of the resource comes from multiple entities in your domain model, or multiple rows in your database, or from reports produced by other services... these are all implementation details. The HTTP transfer of documents over a network application doesn't care.
That also means that we can have multiple resources that include the same information in their representations. Think "pages in wikipedia" that duplicate each others' information.
Resource identifiers on the web are semantically opaque. All three of these identifiers are understood to be different resources
/A
/A?enriched
/B
We human beings looking at these identifiers might expect /A?enriched to be semantically closer to /A than /B, but the machines don't make that assumption.
It's perfectly reasonable for /A?enriched to produce representations using a different schema, or even a different content-type (as far as the HTTP application is concerned, it's perfectly reasonable that /A be an HTML document and /A?enriched be an image).
Because the machines don't care, you've got additional degrees of freedom in how you design both you resources and your resource identifiers, which you can use to enjoy additional benefits, including designing a model that's easy to implement, or easy to document, or easy to interface with, or easy to monitor, or ....
Design is what we do to get more of what we want than we would get by just doing it.
I need to query a property of Saga Data class to get a list. It is stored on SqlPersistance table [Data] column as a serialized object.
Think about a scenario that my SagaData has a property called UserName, so I want to query every saga related to that user.
In a sloppy way, I can query the column content, get the list and can create Saga objects out of the content, by querying like:
SELECT [Id]
,[Correlation_TaskId]
,[Metadata]
,[Data]
,[PersistenceVersion]
,[SagaTypeVersion]
,[Concurrency]
FROM [myWonderfulDb].[dbo].[MyWonderfulPeristanceTable]
where JSON_VALUE(Data,'$.Username') = 'arthur'
but I am looking for an elegant way to do it by possibly using NserviceBus API's.
There is a SagaFinder implementation described in ParticularSoftware documentation (link: https://docs.particular.net/persistence/sql/saga-finder) but this returns only one object which does not perfectly fit into my scenario.
Here how it is implemented in the documentation:
class SqlServerSagaFinder :
IFindSagas<MySagaData>.Using<MyMessage>
{
public Task<MySagaData> FindBy(MyMessage message, SynchronizedStorageSession session, ReadOnlyContextBag context)
{
return session.GetSagaData<MySagaData>(
context: context,
whereClause: "JSON_VALUE(Data,'$.PropertyPathInJson') = #propertyValue",
appendParameters: (builder, append) =>
{
var parameter = builder();
parameter.ParameterName = "propertyValue";
parameter.Value = message.PropertyValue;
append(parameter);
});
}
}
Any ideas appreciated. Thank you!
We have guidance about querying saga state available at
https://docs.particular.net/nservicebus/sagas/#querying-saga-data
In short, you can query the saga data, there is no out of the box way provided by NServiceBus because we recommend using a different approach instead:
the saga to publish events containing the required data and have handlers that process these events and store the data in one or more read model(s) for querying purposes.
The major points why we don't recommend it is in my viewpoint these two
By exposing the data outside of the safeguards of the business logic in the saga the risk is that the data is not treated as
read-only. Eventually, a component tries to bypass the saga and
directly modify the data.
Querying the data might require additional indexes, resources etc. which need to be managed by the component issuing the query.
Those additional resources can influence saga performance.
The only purpose of the SagaFinder is to find a single instance of a certain saga that correlates with the incoming message. I'm not aware of any framework functionalities to query saga (data) instances.
Using Camunda as the tool for orchestration of the microservices. At later time, I find the process_instances_id generated necessary for continuing a particular process by using it in messageEventReceived(). Code as follows:
val processid = getProcessID(key1, key2)
val runtimeService = processengine.getRuntimeService
val subscription = runtimeService.createEventSubscriptionQuery
.eventType("message")
.eventName(eventname)
.processInstanceId(executionid)
.singleResult
runtimeService.messageEventReceived(subscription.getEventName, subscription.getExecutionId)
As of this moment the processid is saved and then retrieved from the database using the getProcessID(...) function when necessary. Is this proper?
Does camunda already have the list of process_ids in its own database? If so, how do I retrieve a particular process instance id just giving composite key(s)? Is that even possible?
It is the common way. You can also use the public api to get the process instance and his id via the process definition key.
See the following example from the documentation:
runtimeService.createProcessInstanceQuery()
.processDefinitionKey("invoice")
.list();
For your given example there is also a simpler way. It is possible to correlate the message via the runtime service.
See this example from the documenation:
runtimeService.createMessageCorrelation("messageName")
.processInstanceBusinessKey("AB-123")
.setVariable("payment_type", "creditCard")
.correlate();
You can use
runtimeService.createProcessInstanceQuery().list();
the query supports fluent criteria for filtering, for example on process_key, variables, businessKey ...
Question
How do I return different results for the same resource?
Details
I have been searching for some time now about the proper way to build a RESTful API. Tons of great information out there. Now I am actually trying to apply this to my website and have run into a few snags. I found a few suggestions that said to base the resources on your database as a starting point, considering your database should be structured decently. Here is my scenario:
My Site:
Here is a little information about my website and the purpose of the API
We are creating a site that allows people to play games. The API is supposed to allow other developers to build their own games and use our backend to collect user information and store it.
Scenario 1:
We have a players database that stores all player data. A developer needs to select this data based on either a user_id (person who owns the player data) or a game_id (the game that collected the data).
Resource
http://site.com/api/players
Issue:
If the developer calls my resource using GET they will receive a list of players. Since there are multiple developers using this system they must specify some ID by which to select all the players. This is where I find a problem. I want the developer to be able to specify two kinds of ID's. They can select all players by user_id or by game_id.
How do you handle this?
Do I need two separate resources?
Lets say you have a controller name 'Players', then you'll have 2 methods:
function user_get(){
//get id from request and do something
}
function game_get(){
//get id from request and do something
}
now the url will look like: http://site.com/api/players/user/333, http://site.com/api/players/game/333
player is the controller.
user/game are the action
If you use phil sturgeon's framework, you'll do that but the url will look like:
http://site.com/api/players/user/id/333, http://site.com/api/players/game/id/333
and then you get the id using : $this->get('id');
You can limit the results by specifying querystring parameters, i.e:
http://site.com/api/players?id=123
http://site.com/api/players?name=Paolo
use phil's REST Server library: https://github.com/philsturgeon/codeigniter-restserver
I use this library in a product environment using oauth, and api key generation. You would create a api controller, and define methods for each of the requests you want. In my case i created an entirely seperate codeigniter instance and just wrote my models as i needed them.
You can also use this REST library to insert data, its all in his documentation..
Here is a video Phil threw together on the basics back in 2011..
http://philsturgeon.co.uk/blog/2011/03/video-set-up-a-rest-api-with-codeigniter
It should go noted, that RESTful URLs mean using plural/singular wording e.g; player = singular, players = all or more than one, games|game etc..
this will allow you to do things like this in your controller
//users method_get is the http req type.. you could use post, or put as well.
public function players_get(){
//query db for players, pass back data
}
Your API Request URL would be something like:
http://api.example.com/players/format/[csv|json|xml|html|php]
this would return a json object of all the users based on your query in your model.
OR
public function player_get($id = false, $game = false){
//if $game_id isset, search by game_id
//query db for a specific player, pass back data
}
Your API Request URL would be something like:
http://api.example.com/player/game/1/format/[csv|json|xml|html|php]
OR
public function playerGames_get($id){
//query db for a specific players games based on $userid
}
Your API Request URL would be something like:
http://api.example.com/playerGames/1/format/[csv|json|xml|html|php]