Redis has a key update notification system but I don't see a way to tell it to call an API endpoint when a key is updated. It seems more like a pub/sub setup. In my case, that API endpoint is a serverless Google Cloud Function. Can redis do this?
I've seen How to implement "trigger" for redis datastore?, but that recommends pub/sub with a subscriber continuously running to receive change messages.
What I'm looking for is for redis itself to call an external HTTP endpoint when something happens. Specifically, a serverless Google Cloud function, which isn't running all the time. The function only "wakes up" on a request so it can't be a Redis subscriber. Hence my question about calling arbitrary HTTP endpoints.
Related
My spring boot application needs to call a third party API to verify an user data. now the API provided from third party, they have some restrictions like they only accept if the request comes from some specific IP.
Now our network consultant has made one of my worker nodes to be able to request this API. I can curl the request to the API and get a proper response. I can deploy my application on that worker node outside of kubernetes using tomcat and get proper response.
But when I deploy it on kubernetes cluster, it does not work. Does not work means that API don't accept/process the request. 503 service not available something like that.
Then I tried to curl the request from inside the pods of worker node and found it does not work.
So, I am guessing my Kubernetes is not using the configuration on the system network.
So is there any way I can make my worker node(even one worker node) to use the system network while calling that or any third party API, as I can see I can request and get proper response from API when I request it outside the cluster but from the worker node pc.
We are making a cloud native enterprise business application on dotnet core mvc platform. The dotnet core default api gateway between frontend application and backend microservices is Ocelot used in Async mode.
We have been suggested to use RabbitMQ message broker instead of Ocelot. The reasoning given for this shift is asynchronous request - response exchange between frontend and microservices. Here, we would like to declare that our application would have few hundred cshtml pages spanning over several frontend modules. We are expecting over thousand users concurrently using the application.
Our concern is that, is it the right suggestion or not. Our development team feels that we should continue using Ocelot api gateway for general request - response exchange between frontend and microservices and use RabbitMQ only for events which are going to trigger backgroup processing and respond after a delay when the job gets completed.
In case you guys feel that yes we can replace Ocelot, then our further concerns about reliable session based request and response. We should not have to programmaticaly corelate response to session requests. Here it may please be noted that with RabbitMQ we are testing with dotnet core MassTransit library. The Ocelot API Gateway is designed to handle session based request-response commnunication.
In RabbitMQ should we make reply queue for each request or should the client maintain a single reply queue for all requests. Should the reply queue be exclusive or durable.
Can single reply queue per client be able to serve to all request or will it be correct to make multiple receive endpoint based on application modules/cshtml pages to serve all our concurrent users with efficient way.
Thanking you all, we eagerly wait for your replies.
I recommend to implement RabbitMQ. You might need to change ocelot to rabbit mq.
As I have previously mentioned, I am using ServiceStack Messaging API (IMessageQueueClient.Publish) as well as the more low-level IRedisClient.PublishMessage.
I use the Messaging API when I need a specific message/request to be processed by only one instance of a module/service, so even though I might have several modules running that all listens for MyRequest, only one service receives the message and processes it.
I use the IRedisClient.PublishMessage when I do a broadcast, a pub/sub situation, sending a request that everyone should receive that listens on that specific Redis channel.
However, I am in a situation where it would be useful to use the Messaging API, but do a broadcast, so that all instances that are listening to a specific message type, gets the message, not just the one.
(The reason for this is to streamline our usage of Redis and how we subscribe to events/request, but I will not get into details about this now. A little more background on this is here.)
Is there a "broadcast way" for the Messaging API?
No, the purpose of ServiceStack Messaging is simply to invoke ServiceStack Services via MQ. Any other MQ features is outside the purpose & scope of ServiceStack MQ, you'd need to instead develop against the MQ Provider APIs directly to access their broadcast features.
Server Events is a ServiceStack feature that supports broadcasting messages to subscribers of user-defined channels, but its a completely different implementation that serves a different use-case for sending "server push" real-time events over HTTP or gRPC, e.g. it doesn't use MQ brokers and pub/sub messages aren't persistent (i.e. only subscribers at time messages are sent will receive them).
For my deployment I have a number of 3rd party systems that can only send HTTP POST requests with metrics (I need in the queue) and they cannot be re-configured. My goal is to have specific endpoints (or vhosts) that when POST'd to will automatically route to the correct queue, without needing the necessary routing key and other standard rabbitmq JSON data. As this modification is not possible in the 3rd party systems.
I can't find any way to do this natively as of now, but I believe it may be possible to configure a HTTP reverse proxy in the front, whereby any data sent to the specific endpoint, will be re-directed to the correct rabbitMQ HTTP endpoint, where I could then bolt in the nessary JSON data so it can be parsed by rabbitmq and placed in the realvent queue. I wanted to check if this is the only logical solution to this, or am I missing something obvious that can be done within rabbitmq's administration page or via config files.
Problem description
i am working on a Xamarin application that consumes a REST API written in Python flask.
The Xamarin application offers virtual shopping lists where user can collaborate on buying stuff they have on a shared list.
To improve the user experience, i want to be able to actively notify the user about finished items on the list.
Possible solutions:
Synchronous API polling from client side
Notifications are stored by the API in a relational database and have a flag indicating if the user received the notification already.
The API has an endpoint GET /users/:user_id/notifications/ that queries the database for notifications and returns a JSON response with those.
Advantages
fairly simple to implement
Problems
synchronous polling creates a huge amount of http requests
API service remains stateless, making a horizontal scaling with a loadbalancer easier
Websocket endpoint on the API
The API has an endpoint POST /users/:user_id/notifications/register which creates a websocket connection between client and API.
The connection is stored to a global array in which each entry maps a client id to a websocket connection.
When a new notification is created, the endpoint makes a lookup in the connection dictionary by comparing the owner id of the notification with the dictionary entries. The notification is sent to appropriate user through the websocket.
Notifications are stored in the database like in the first approach.
When a user calls the endpoint, a new websocket connection will be established first and upon success the API sends all unseen notifications from the database to the user.
Advantages
API can push notifications to clients asynchronously
Problems
When a user terminates the websocket connection his dictionary entry will persis
Retaining one websocket connection per user permanently adds additional overhead to the API
Horizontal scalability of the API is more difficult because the service is not stateless anymore (Websocket connection information saved in
RabbitMQ
The API uses a RabbitMQ service to send notifications to the client. Every client uses subscribes to his own notification queue to prevent the broadcasting of messages.
Advantages
API remains stateless
Problems
Notifications needs to be resend to the exchange when a user is offline
Amount of queues grows drastically
Additional costs for RabbitMQ service
High temporary load on the RabbitMQ service when many users come online in the same time
Final words
It would be interesting to hear the opinion of others.
I believe the active distribution of notifications from backen services to clients i a very common use case.
best,
D
I would use RabbitMQ and consume events forwarding them as push notifications. This will work while the user is not actively connected to the website and enhance the engagement with each user experience that will return to the website when notified for more information see How to setup basic web push notification functionality using a Flask backend or How to send push notifications to a browser in ASP.NET Core or Sending Notifications with Spring Boot, Angular, and Firebase Cloud Messaging this way the RabbitMQ will not wait until the user is back online. If the user is online you can forward the notification directly to the Xamarin application via WebSockets and a load balancer like NGINX that can handle many WebSockets in an optimized way.
Synchronous API polling from the client-side is the less preferred way since it overloads the webserver with requests while nothing was changed.
I don't think the scalability of WebSocket is a problem. You can scale up easily with pub/sub. The hotspot of long connections is a kind of serious problem.
For one-way communication, I would suggest Server sent event. In the end, it usually depends on what your team is confident with.
I can recommend on a different approach for API that provides JSON which is called GraphQL
It supports subscriptions capabilities that are pushed by the GraphQL API Server (using web sockets)
GraphQL is considered today to be better than RESTful API since its very flexible and you can get exactly the data you need with one query.