I'd like to subscribe to multiple event channels using pattern matching.
node-redis client:
import { createClient } from 'redis';
const subscriber = createClient(config);
subscriber.config("SET", "notify-keyspace-events", "KEx$");
subscriber.psubscribe('__keyevent#0__:set:expired');
redis-cli:
psubscribe '__keyevent#0__:set:expired'
But it doesn't work. So I wonder which subscription pattern should be used to match multiple event channels. Would it be a best practice to use multi channels through args in subscribe instead? Like this: subscribe('__keyevent#0__:set', '__keyevent#0__:expired').
Related
i'm currently working about Redis Expire Event
My goal : get the Value, Field to do something in next process after Data in Redis already expire
so i had found Redis Keyspace Notifications Feature
That Allow client to Subscribe to Channel in Redis to Recieve Event affecting Data in reset like Expire
so i have some example code : https://github.com/toygame/nodejs-redis-keyspace-notifications
subscriber.subscribe("__keyevent#0__:expired")
subscriber.on('message', async(channel, message) => {
// do somethings
console.log(message);
})
Result : Key0
And this work find but the Result i got is only Key that i have set into redis and expired
I have already do some research
https://medium.com/nerd-for-tech/redis-getting-notified-when-a-key-is-expired-or-changed-ca3e1f1c7f0a
but its found only Event that maybe i can get as result but not for Value, Field that i expect
is their anyway to get Those Value and Field ?
FYI. Document https://redis.io/topics/notifications
UPDATE
according to this https://stackoverflow.com/a/42556450/11349357
Keyspace notifications do not report the value, only the key's name and/or command performed are included in the published message.The main underlaying reasoning for this is that Redis values can become quite large.
If you really really really need this kind of behavior, well that's pretty easy actually. Because keyspace notifications are using Pub/Sub messages, you can just call PUBLISH yourself after each relevant operation, and with the information that you're interested in.
look like i can't use this Redis Keyspace but i have to publish its on my own
You can use RedisGears to process keyspace notification and get both key and value.
You can write your processing code in python and register it in Redis.
e.g. Capture each keyspace event and store to a Stream
GearsBuilder() \
.foreach(lambda x: execute('XADD', "notifications-stream", '*', *sum([[k,v] for k,v in x.items()],[]))) \
.register(prefix="person:*", eventTypes=['hset', 'hmset'], mode='sync')
You can read more about this example here: https://oss.redis.com/redisgears/examples.html#reliable-keyspace-notification
Let's say I have a situation where I need to wait for up to 1 minute for some action to be performed.
If it is expired, then try different action.
My current solution proposal is based on RabbitMQ features.
I would create following resources:
#Bean
DirectExchange exchangeDirect() {
return new DirectExchange("exchange.direct");
}
#Bean
Queue bufferQueue() {
Map<String, Object> args = new HashMap<>();
args.put("x-message-ttl", amqpProperties.getTimeToLive().toMillis());
args.put("x-dead-letter-exchange", "exchange.direct");
args.put("x-dead-letter-routing-key", "timedOutQueue");
return new Queue("buffer.queue", true, false, false, args);
}
#Bean
Queue timedOutQueue() {
return new Queue("timed.out.queue", true);
}
#Bean
Binding bufferQueueToExchangeDirect() {
return bind(bufferQueue())
.to(exchangeDirect())
.with("buffer.queue");
}
#Bean
Binding timedOutQueueToExchangeDirect() {
return bind(timedOutQueue())
.to(exchangeDirect())
.with("timed.out.queue");
}
When I add action to bufferQueue and I don't receive any delivery update within 1 minute, this request is then moved to timedOutQueue thanks to bufferQueue's TTL.
I can attach application rabbit listener to timedOutQueue and use different action.
When I add action to bufferQueue and I receive confirmation that action was successfully performed, I'd like to remove given action event from bufferQueue.
I couldn't find such feature in RabbitMQ, i.e. being able to receive selectively.
I also found some articles saying that selective consuming is antipattern.
Is it possible to selectively consume messages from RabbitMQ queue?
What is proper way to implement this pattern in RabbitMQ?
There is no concept of message selection in RabbitMQ.
The "proper" way for an application that wants to selectively receive messages is to use multiple queues/routing keys with a consumer on each specific queue he expresses interest in.
However, there is no way to "remove" a message from the middle of a queue; only the head.
When I add action to bufferQueue and I receive confirmation that action was successfully performed, I'd like to remove given action event from bufferQueue.
That makes no sense to me; when the message timed out in bufferQueue due to TTL, and was moved to timedOutQueue, it no longer exists in bufferQueue so there is nothing to remove.
There is also no mechanism to ...
and I don't receive any delivery update within 1 minute,
... because each message in a queue is independent.
It doesn't sound like your application is suitable for a message broker at all.
I am using Jedis java client for redis. My requirement is that when someone add item to list, say mylist by doing jedisClient.lpush("mylist", "this is my msg"), I need to get notification.
Is this possible ?
Yes, it is possible to achieve that in one of two ways.
The first approach is to use Redis' keyspace notifications. Configure Redis to generate list events with the following configuration directive:
CONFIG SET notify-keyspace-events El
Then, subscribe to the relevant channel/channels. If you want to subscribe only to mylist's changes, do:
SUBSCRIBE __keyevent#0__:mylist
Or, use PSUBSCRIBE and listen to events to matching key names that match a pattern.
Note, however, that keysapce notifications will not provide the actual pushed value. You can use Lua scripts as an alternate approach and implement your own notifications mechanism. For example, use the following script to push and publish a custom message to a custom channel:
local l = redis.call("LPUSH", KEYS[1], ARGS[1])
redis.call("PUBLISH", "mylistnotif:" .. KEYS[1], "Pushed value " .. ARGS[1])
return l
Make sure that "someone" uses that script to do the actual list-pushing and subscribe to the relevant channel/channels.
I am having a great experience with ServiceStack & Redis, but I'm confused by ThreadPool and Pub/Sub within a thread, and an apparent limitation for accessing Redis within a message callback. The actual error I get states that I can only call "Subscribe" or "Publish" within the "current context". This happens when I try to do another Redis action from the message callback.
I have a process that must run continuously. In my case I can't just service a request one time, but must keep a thread alive all the time doing calculations (and controlling these threads from a REST API route is ideal). Data must come in to the process on a regular basis, and data must be published. The process must also store and retrieve data from Redis. I am using routes and services to take data in and store it in Redis, so this must take place async from the "calculation" process. I thought pub/sub would be the answer to glue the pieces together, but so far that does not seem possible.
Here is how my code is currently structured (the code with the above error). This is the callback for the route that starts the long term "calculation" thread:
public object Get(SystemCmd request)
{
object ctx = new object();
TradingSystemCmd SystemCmd = new TradingSystemCmd(request, ctx);
ThreadPool.QueueUserWorkItem(x =>
{
SystemCmd.signalEngine();
});
return (retVal); // retVal defined elsewhere
}
Here is the SystemCmd.signalEngine():
public void signalEngine(){
using (var subscription = Redis.CreateSubscription())
{
subscription.OnSubscribe = channel =>
{
};
subscription.OnUnSubscribe = channel =>
{
};
subscription.OnMessage = (channel, msg) =>
{
TC_CalcBar(channel, redisTrade);
};
subscription.SubscribeToChannels(dmx_key); //blocking
}
}
The "TC_CalcBar" call does processing on data as it becomes available. Within this call is a call to Redis for a regular database accesses (and the error). What I could do would be to remove the Subscription and use another method to block on data being available in Redis. But the current approach seemed quite nice until it failed to work. :-)
I also don't know if the ThreadPool has anything to do with the error, or not.
As per Redis documentation:
Once the client enters the subscribed state it is not supposed to
issue any other commands, except for additional SUBSCRIBE, PSUBSCRIBE,
UNSUBSCRIBE and PUNSUBSCRIBE commands.
Source : http://redis.io/commands/subscribe
I am fairly new to Scala and I'm working on an application (library) which is a client to a 3rd-party service (I'm not able to modify the server side and it uses custom binary protocol). I use Netty for networking.
I want to design an API which should allow users to:
Send requests to the server
Send requests to the server and get the response asynchronously
Subscribe to events triggered by the server (having multiple asynchronous event handlers which should be able to send requests as well)
I am not sure how should I design it. Exploring Scala, I stumble upon a bunch of information about Actor model, but I am not sure if it can be applied there and if it can, how.
I'd like to get some recommendations on the way I should take.
In general, the Scala-ish way to expose asynchronous functionality to user code is to return a scala.concurrent.Future[T].
If you're going the actor route, you might consider encapsulating the binary communication within the context of a single actor class. You can scale the instances of this proxy actor using Akka's router support, and you could produce response futures easily using the ask pattern. There are a few nice libraries (Spray, Play Framework) that make wrapping e.g. a RESTful or even WebSocket layer over Akka almost trivial.
A nice model for the pub-sub functionality might be to define a Publisher trait that you can mix in to some actor subclasses. This could define some state to keep track of subscribers, handle Subscribe and Unsubscribe messages, and provide some sort of convenient method for broadcasting messages:
/**
* Sends a copy of the supplied event object to every subscriber of
* the event object class and superclasses.
*/
protected[this] def publish[T](event: T) {
for (subscriber <- subscribersFor(event)) subscriber ! event
}
These are just some ideas based on doing something similar in some recent projects. Feel free to elaborate on your use case if you need more specific direction. Also, the Akka user list is a great resource for general questions like this, if indeed you're interested in exploring actors in Scala.
Observables
This looks like a good example for the Obesrvable pattern. This pattern comes from the Reactive Extensions of .NET, but is also available for Java and Scala. The library is provided by Netflix and has a really good quality.
This pattern has a good theoretical foundation --- it is the dual to the iterator in the category theoretical sense. But more important, it has a lot of practical ideas in it. Especially it handles time very good, e.g. you can limit the event rate you want to get.
With an observable you can process events on avery high level. In .NET it looks a lot like an SQL query. You can register for certain events ("FROM"), filter them ("WHERE") and finally process them ("SELECT"). In Scala you can use standard monadic API (map, filter, flatMap) and of course "for expressions".
An example can look like
stackoverflowQuestions.filter(_.tag == "Scala").map(_.subject).throttleLast(1 second).subscribe(println _)
Obeservables take away a lot of problems you will have with event based systems
Handling subsrcriptions
Handling errors
Filtering and pre-processing events
Buffering events
Structuring the API
Your API should provide an obesrvable for each event source you have. For procedure calls you provide a function that will map the function call to an obesrvable. This function will call the remote procedure and provide the result through the obeservable.
Implementation details
Add the following dependency to your build.sbt:
libraryDependencies += "com.netflix.rxjava" % "rxjava-scala" % "0.15.0"
You can then use the following pattern to convert a callback to an obeservable (given your remote API has some way to register and unregister a callback):
private val callbackFunc : (rx.lang.scala.Observer[String]) => rx.lang.scala.Subscription = { o =>
val listener = {
case Value(s) => o.onNext(s)
case Error(e) => o.onError(o)
}
remote.subscribe(listener)
// Return an interface to cancel the subscription
new Subscription {
val unsubscribed = new AtomicBoolean(false)
def isUnsubscribed: Boolean = unsubscribed.get()
val asJavaSubscription: rx.Subscription = new rx.Subscription {
def unsubscribe() {
remote.unsubscribe(listener)
unsubscribed.set(true)
}
}
}
If you have some specific questions, just ask and I can refine the answer
Additional ressources
There is a very nice course from Martin Odersky et al. at coursera, covering Observables and other reactive techniques.
Take a look at the spray-client library. This provides HTTP request functionality (I'm assuming the server you want to talk to is a web service?). It gives you a pretty nice DSL for building requests and is all about being asynchronous. It does use the akka Actor model behind the scenes, but you do not have to build your own Actors to use it. Instead the you can just use scala's Future model for handling things asynchronously. A good introduction to the Future model is here.
The basic building block of spray-client is a "pipeline" which maps an HttpRequest to a Future containing an HttpResponse:
// this is from the spray-client docs
val pipeline: HttpRequest => Future[HttpResponse] = sendReceive
val response: Future[HttpResponse] = pipeline(Get("http://spray.io/"))
You can take this basic building block and build it up into a client API in a couple of steps. First, make a class that sets up a pipeline and defines some intermediate helpers demonstrating ResponseTransformation techniques:
import scala.concurrent._
import spray.can.client.HttpClient
import spray.client.HttpConduit
import spray.client.HttpConduit._
import spray.http.{HttpRequest, HttpResponse, FormData}
import spray.httpx.unmarshalling.Unmarshaller
import spray.io.IOExtension
type Pipeline = (HttpRequest) => Future[HttpResponse]
// this is basically spray-client boilerplate
def createPipeline(system: ActorSystem, host: String, port: Int): Pipeline = {
val httpClient = system.actorOf(Props(new HttpClient(IOExtension(system).ioBridge())))
val conduit = system.actorOf(props = Props(new HttpConduit(httpClient, host, port)))
sendReceive(conduit)
}
private var pipeline: Pipeline = _
// unmarshalls to a specific type, e.g. a case class representing a datamodel
private def unmarshallingPipeline[T](implicit ec:ExecutionContext, um:Unmarshaller[T]) = (pipeline ~> unmarshal[T])
// for requests that don't return any content. If you get a successful Future it worked; if there's an error you'll get a failed future from the errorFilter below.
private def unitPipeline(implicit ec:ExecutionContext) = (pipeline ~> { _:HttpResponse => () })
// similar to unitPipeline, but where you care about the specific response code.
private def statusPipeline(implicit ec:ExecutionContext) = (pipeline -> {r:HttpResponse => r.status})
// if you want standard error handling create a filter like this
// RemoteServerError and RemoteClientError are custom exception classes
// not shown here.
val errorFilter = { response:HttpResponse =>
if(response.status.isSuccess) response
else if(response.status.value >= 500) throw RemoteServerError(response)
else throw RemoteClientError(response)
}
pipeline = (createPipeline(system, "yourHost", 8080) ~> errorFilter)
Then you can use wrap these up in methods tied to specific requests/responses that becomes the public API. For example, suppose the service has a "ping" GET endpoint that returns a string ("pong") and a "form" POST endpoint where you post form data and receive a DataModel in return:
def ping()(implicit ec:ExecutionContext, um:Unmarshaller[String]): Future[String] =
unmarshallingPipeline(Get("/ping"))
def form(formData: Map[String, String])(implicit ec:ExecutionContext, um:Unmarshaller[DataModel]): Future[DataModel] =
unmarshallingPipeline(Post("/form"), FormData(formData))
And then someone could use the API like this:
import scala.util.{Failure, Success}
API.ping() foreach(println) // will print out "pong" when response comes back
API.form(Map("a" -> "b") onComplete {
case Success(dataModel) => println("Form accepted. Server returned DataModel: " + dataModel)
case Failure(e) => println("Oh noes, the form didn't go through! " + e)
}
I'm not sure if you will find direct support in spray-client for your third bullet point about subscribing to events. Are these events being generated by the server and somehow sent to your client outside the scope of a specific HTTP request? If so, then spray-client will probably not be able to help directly (though your event handlers could still use it to send requests). Are the events occurring on the client side, e.g. the completion of deferred processing initially triggered by a response from the server? If so, you could actually probably get pretty far just by using the functionality in Future, but depending on your use cases, using Actors might make sense.