I have a microservice that among other things is used as a "caching proxy" (I'm not sure that this term is correct). It is in between the application API and Azure API. This microservice periodically fetches some data from Azure for several resources and stores it in Redis. Application API from the other side requests the resource data but reads it not from Azure itself, but from Redis.
(This is done in order to limit the scale of requests hitting the Azure API when having a high load on the application API.)
The periodical polling is currently implemented as a naive "while not canceled - fetch, update Redis and sleep for 15 seconds".
This worked well while I had only one instance of the microservice. But now due to new requirements, I have an automatic scaling of my microservice. And that means that if there are 5 instances of the microservice running right now - I'm hitting the Azure API 5 times more frequently than I should.
My question is how can I fix this to do "one request to Azure API per resource once in 15 seconds" - no matter how many microservice instances I have?
My constraints are:
do the minimal changes since the microservice is already in Production;
use the existing resources as much as possible (apart from Redis the microservice is already using message queues - Azure Service Bus).
Ideas I have:
make only one instance a "master" - only this instance will fetch data from Azure. But what should I do when auto-scaling shuts this instance down? How can I detect this and decide on a new master instance? Maybe I could store the master instance identifier in a short-living key in Redis and prolong it every time the resource data is retrieved from Azure? If there is no key in Redis - a new master instance is selected.
use Azure Service Bus message scheduling - on microservice application startup the instance schedules a message in the next 15 seconds which will be received by only one microservice instance. On receiving this message the microservice instance will fetch the data from Azure, update Redis - and schedule another message in the next 15 seconds. This time another microservice instance can receive the instance and do the same - fetch data, update Redis, and schedule the next message. But I don't know how to avoid parallel message chains initiated when several microservice instances are started/restarted.
Anyway, I don't see any good solution for my problem and would appreciate a hint.
Related
Using ASP.NET Core microservices, both API and worker roles, running in Azure Service Fabric.
We use Service Bus to do inter-microservice communication.
Consider the following situation;
Each microservice holds a local, in-mem copy of cached objects of type X.
One worker role is responsible for processing a message that would result in a rebuild of this cache for all instances.
We are having multiple nodes, and thus multiple instances of each microservice in Service Fabric.
What would be the best approach to trigger this update?
I though of the following approaches:
Calling SF for all service replica's and firing an HTTP POST on each replica to trigger the update
This however does not seem to work as worker roles don't expose any APIs
Creating a specific 'broadcast' topic where each instance registers a subscription for, and thus using pub/sub mechanism
I fail to see how I can make sure each instance has it's own subscription, but also I don't end up with ghost subscriptions when something happens like a crash
You can use the OSS library Service Fabric Pub Sub for this.
Every service partition can create its own subscription for messages of a given type.
It uses the partition identifier for subscriptions, so crashes and moves won't result in ghost subscriptions.
It uses regular SF remoting, so you won't need to expose API's for messaging.
Microservice architecture and sharing common application data.
Scenario being:
There are today 17 microservices for some online social media service and 9 of them need to know who is connected to who in order for their function to work. To prevent each service constantly asking the "authentication" or "connections" microserice for the list, all services register to recieve a copy of the connections per user and store in a cache.
A proposal for the mechanism to deliver the data, or instruction to fetch data could be rabbitmq.
However, each microservice is a cluster of docker containers orchestrated by k8s for scalability.
Each container registers to listen to a collection of exchanges they are interested in... so for the "news feed" service that could be say 5 connections...
Below is an illustration of the proposed setup:
T1 - user A accept a friend request
T2 - The connections service (MS1) makes the connection in its primary database
T3 - MS1 published to a rabbitmq exchange the said event
T4 - rabbitmq exchange emits to all Q's (ie all other microservices registered)
T5 - All the nodes within the MS2 cluster pickup the event and act... their action (in this case) will be to update the cache of the friend connections.
T6 - user A requests the data for their newsfeed, MS2 now queries its database with the use of its local cache
This is all good:
The connection service didn't know or care who got the data, only that it should emit to 1 exchange via the single rabbitmq entry point
The developer of MS2 only needed to know about the location of the rabbitmq instance
The developer of all the other services the same.. they handle the data in their own brilliant way.
The 1 exception is.. there were 3 instances of MS2 so that would be 3 database writes.. if the system scales to 10 that would be 10 db writes etc etc.
Question
How is this problem bypassed... how to ensure only 1 of the MS2 instances will act?
Should the newsfeed microservice be delivered with its own internal q system to manage the data from the exchange? Is it possible to route all the messages via the load balancer so only 1 instance of MS2 gets a message? I don't want to start managing lots of lots of queues by hand as this will be a pain and defeat the simplicity of the exchange design.
So, all instances if M2 will share a queue and work using the competing consumer pattern, every messages is consumed once and if all instance of M2 goes down the queue grows until they come back up again.
M2, M3 and M4 will each create ONE queue for what M1 publishes.
Let's name them them
M2_from_M1, M3_from_M1 and M4_from_M1.
They will all also create a binding against the exchange M1 uses and on the routing key for this message.
Now, instances of M2 will all consume from M2_from_M1, instances of M3 will all consume from M3_from_M1 and so on.
If all instances of any of these are down it's queue will start to fill up but that is fine since it will be consumed later.
Regarding the overall architecture. Try first with actually making the call between M2 and M1, access time between pods is probably very fast and you could probably cache both in M1 and in M2 for a while. Worst outcome is that you see news from people you no longer follow, or that you don't get news from new contacts.
I've read other questions on StackOverflow regarding using NSB on SF and also the sample on github (outdated) and I'm still not sure how to configure NServiceBus properly for this platform.
I'm looking to set up a send only publish/subscribe workflow. What I can't determine through my research is how to set this up so that only one instance of a particular service responds to the message.
For example: 3 services running on the standard 5 nodes (so pretend 5 instances of each of the 3 services).
Existing load balancer routes an http request to a specific instance of Service A.
Service A publishes the "OrderComplete" event
Services B and C both subscribe to the event.
How can I make sure that only one instance of Services B and C respond instead of all 5 instances of Service B and all 5 instances of Service C?
All the services are currently Stateless services.
I was thinking of using the AzureServiceBus or AzureStorageQueue transport.
Stateless approach is fine. You do not need to go into stateful services with a single partition unless you want to leverage reliable collections for your services. But let's look at both options
Going with Stateless services
It's ok to have multiple instances of your services. Yes, they all will create subscriptions. I'd argue that is exactly what you want - competing consumers. More service instances you have, more throughput you'll get, i.e. handling more messages.
What I can't determine through my research is how to set this up so that only one instance of a particular service responds to the message.
This will happen automatically due to the nature of the competing consumer transport (both ASB and ASQ).
Going with Stateful services
With stateful services you need to be very careful. Yes, you could go with a single partition per service, hence having a single primary replica handling your messages. But then, arguably, you're wasting your cluster resources by not utilizing them for concurrent processing of many messages. If you decide to partition your service, then you won't be able to use reliable collections as replicas of services do not share reliable collections among themselves. Should you choose to use partitioned stateful services w/o reliable collections, well, then you better to utilize stateless counterpart.
Note: NSB will provide support for running with stateful services to take advantage of the reliable collections for persistence needs, but even then partitioning is something that would need to be through through to align with business needs. If you do not have a need like that, I'd suggest to stick to stateless services and Azure Storage persistence.
In the NSB/SF sample on github there is a Stateful service that handles a command. What is important is that in the application it has an PartitionCount=1. The same goes for all other solutions with NSB I have seen, only one partition or instance of each service that handles messages. Otherwise you would end up with one subscription per instance for each message as you describe.
Perhaps you could adopt the Distributor to achieve load balancing between multiple instances of the same service, but afaik Distributor only works with MSMQ so you will have to rewrite it to work with SF and Azure Service Bus.
If you stick with single instances, it should work fine for you. You would still get some benefit from SF as it ensures your services are up and running, but load balancing between multiple instances will require some work for you.
All ,
We have a requirement to develop a azure based platform, in which the user can configure multiple pharmaceutical instruments, start measurements on them and analyze the measured data. The typical components in the azure based platform will be following
1 - A .NET based 4 client application running on the computer connected to each instrument. This client application should receive the start measurement command from the azure platform , perform the measurement and update the result back to the azure*
2 - A set of services[probably REST based] which will get the results from the client application and update the database on the cloud
3 - A set of services and business logic which which can be used to perform analysis on the data
4 - A asp.net web application where the user can view instrument details , start measurement etc
There is a two way communication between the Azure platform and the client application i.e. the client needs to update results to the azure and the azure needs to initiate measurement on the instrument via the client application
In such a scenario , what is the recommended approach for the azure platform to communicate to the clients. Is it any of the following
1 - Create a duplex service between the client and server and provide a call back interface to start the measurement
2 - Create a command queue using Azure message queue for each client. when a measurement needs to be started , a message will the put on the queue. The client app will always read from the queue and execute the command
or do we have any other ways to do this , any help is appreciated
We do not fully understand your scenario and constraints around it, but as pointers, we have seen lot of customers use Azure storage queues to implement master-worker scenario (some component adds message to appropriate queue to get work done (take measurements in your case) and workers polling the queue to process this work (client computer connected to your instrument in this case)).
In terms of storing the results back, your master component could provide SAS access to client to write results back to specific blob in an Azure storage account and either have your service and business logic monitor existence of that blob to start your analysis.
Above approach will decouple your client from server and make communication asynchronous via storage. Again, these are just pointers and you would be the best person to pick the right approach that suits your requirement
For communication between the server and the client, you could use SignalR http://signalr.net/ there are two forms of messaging systems supported "as a service" on Azure, these are Service Bus and Message Queues - see this link http://msdn.microsoft.com/en-us/library/hh767287.aspx
I'm playing around with windows azure and I would like to build a clouded server application that receives messages from many different clients, such as mobile and desktop.
I would like to build the client so that they work while in "offline-mode", i.e. I would like the client to build up a local queue of messages that are sent to the azure server as soon as they get online.
Can I accomplish this using wcf and/or azure queing mechanism, so that I don't have to worry about whether the client is online or offline when I write the code?
You won't need queuing in the cloud to accomplish this. For the client app to be "offline enabled" you need to do queuing on the client. For this there are many options, a local database, xml files, etc. Whenever the app senses network availability you can upload your queue to Azure. And yes, you can use WCF for that.
For the client queue/sync stuff you could take a look at the Sync Framework.
I haven't found a great need for the queue so far. Maybe it's just that I'm not seeing it in my app view. Could also be that the data you can store in the queue is minimal. You basically store short text strings (like record ids), and then you have to do something with the ID when you pull it from the queue, such as look it up, delete it, whatever.
In my app, I didn't use the queue at all, just as Peter suggests. I wrote directly to table storage (accessed via it's REST interface using StorageClient) from the client. If you want to look at a concrete example, take a look at http://www.netalerts.mobi/traffic. Like you, I wanted to learn Azure so I built a small web site.
There's a worker_role that wakes up every 60 seconds. Using one thread, it retrieves any new data from it's source (screen scraping a web page). New entries are stored directly in table storage (no need for a queue). Another thread deletes entries in table storage that are older than a specified threshold (there's no issue with running multiple threads against table storage). And then I'm working on the third thread which is designed to send notifications to handheld devices.
The app itself is a web_role, obviously.