I have several hardware devices that send large amount of data to the cloud. I need to store them on the cloud, process them and send some status reports based on the data analysed to clients who are interested in looking at those results. Clients are smart phone users.
A single client is interested in knowing one or more hardware status report.
I need to make this scalable using Azure, i.e be able to monitor 1000s of hardwares. I need cloud storage , cloud computing power and ability to send data from many hardwares and send reports to clients that are monitoring these hardwares.
I am new to WCF and Azure any guidance on how to write a scalable application using WCF and Azure will be very useful. Please explain how it can be scalable. Do I have to use worker role / web role ? I have some computationally intensive data processing to be done to produce the reports, that clients are interested in.
Shashi
Sounds like an interesting project...
You can host WCF Services in a WCF Service Web Role, which is a web role with starting artifacts for hosting WCF services.
For entensive processing you can use worker roles. When data is received, a WCF service cam place a message on a Service Bus queue, which will be received by a worker role, which can process the data asynchronouslty.
For data storage you could look at the Table and Blob storage in Windows Azure Storage, or look at Windows Azure SQL Database if you need relational storage. There are advantages and disadvanteges to both approaches.
There is quite a lot of technology to evaluate, so it might be worth running through a few tutorials to get an idea of what will make for the best implementation. The Windows Azure Training Kit is a good starting place for this.
http://www.microsoft.com/en-us/download/details.aspx?id=8396
Regards,
Alan
You can scale by increasing the instances of both Web and Worker roles based on the load. Azure roles (cloud service) is a stateless (wont support sticky sessions by default) hence the response of request from same client will be equally distributed to all your instances (round robin).
#coolshashi.
By default one Azure Cloud Solution can consist of 5 different roles (the mix of Web or Worker doesn't matter). Each of those roles can have multiple instances.
For example: 7 Instances of your Web Role could form your front-end Web Farm that places orders on a Service Bus Queue. These orders might be read by 2 instances of your Worker Role which processes them & put them into a database.
The only difference between a web & worker role is that the Web Role has IIS installed & started.
It is easy to configure the number of instances per role to dynamically change based on some metric you define (ie: CPU use or Messages in a Queue). So the solution can scale up to handle load & shrink to save money when not required.
Most Azure subscriptions (or accounts) are initially constrained to 20 cores. This is to prevent you from accidentally creating a massive bill. If your solution requires more, a quick chat to Microsoft can remove that limit to give you as much as you desire.
Related
We are developing a Web API using .Net Core. To perform background tasks we have used Hosted Services.
System has been hosted in AWS Beantalk Environment with the Load Balancer. So based on the load Beanstalk creates/remove new instances of the system.
Our problem is,
Since background services also runs inside the API, When load balancer increases the instances, number of background services also get increased and there is a possibility to execute same task multiple times. Ideally there should be only one instance of background services.
One way to tackle this is to stop executing background services when in a load balanced environment and have a dedicated non-load balanced single instance environment for background services only.
That is a bit ugly solution. So,
1) Is there a better solution for this?
2) Is there a way to identify the primary instance while in a load balanced environment? If so I can conditionally register Hosted services.
Any help is really appreciated.
Thanks
I am facing the same scenario and thinking of a way to implement a custom service architecture that can run normally on all of the instance but to take advantage of pub/sub broker and distributed memory service so those small services will contact each other and coordinate what's to be done. It's complicated to develop yes but a very robust solution IMO.
You'll "have to" use a distributed "lock" system. You'll have to use, for example, a distributed memory cache who put a lock when someone (a node of your cluster) is working on background. If another node is trying to do the same job, he'll be locked by the first lock if the work isn't done yet.
What i mean, if all your nodes doesn't have a "sync handler" you can't handle this kind of situation. It could be SQL app lock, distributed memory cache or other things ..
There is something called Mutex but even that won't control this in multi-instance environment. However, there are ways to control it to some level (may be even 100%). One way would be to keep a tracker in the database. e.g. if the job has to run daily, before starting your job in the background service you might wanna query the database if there is any entry for today, if not then you will insert an entry and start your job.
I would like to create an app where the user can add and view data. Either adding at a desktop/tablet or phone and reading from either source. I would like the data store to be synced between any of the user's devices.
I'm starting to play with the Trial of Azure, and it looks promising. Probably a solid way to sync data through to cloud between users' devices. Other than syncing between a users devices, I have no need for cloud services currently.
I've seen some apps that do a 'Backup/Restore' model with the user's SkyDrive account. But this seems to be a manual process. I'd like to see it done seamlessly.
I've looked into Sync services, but that would be more like a hub/spoke solution. Again, I don't need a central database.
What are some options? At this point, I would be fine using just Windows 8 patterns/practices.
Because they are separate devices, you will need to have some service layer to do the store/forward for you. With that you have two basic choices, use the end user's own storage (aka SkyDrive) or use your own storage (aka Windows Azure).
SkyDrive is fully supported through the Live SDKs and provides a secure way to allow a user to share store their data, and synchronize it across multiple devices. You get security, and there is no cost for the server side storage on your part. The user owns their storage, not you. The limitation is that you may have issues sharing that same data across other devices or users where SkyDrive (or whatever file sync service you use) is not available.
With a service layer, like Azure, you have a lot more flexibility, but you also will be responsible for maintaining (and paying for) that server side storage / services. Have you looked at "Windows Azure Mobile Services". With your Azure account you get 10 free Azure Mobile Services. You will pay for the SQL data storage on the backend, and that cost will depend on the amount of data you store on the server side. You also need to make sure to architect your application in a way to protect an individual users' data, but it is actually pretty easy to do, well documented, and gives you a lot of options.
Lastly, you may consider what type of data you want to share. SkyDrive is great for "Files". Pics, Songs, Videos, etc. Windows Azure Mobile Services (WAMS) is great for "Data".
Neither model is right or wrong. It just depends on your goals.
Hope that helps you go through the thought process
I have created a WCF web service that will upload data from SQL Server to our ISeries. When an end user is finished with their data entry (a batch), they will "send" the batch number to the web service. The web service will then upload that data to the ISeries. It cannot be assumed that this will be a quick process and there may be many end users hitting the web service at once. Likewise, because the way the database is setup on the ISeries, I can't upload this data simultaneously because we may run into locks, misfired triggers, etc. So, I want somehow to queue these calls so that they are done in order received.
I have been searching for methods to do this and there's a lot of information in 2011 and earlier discussing MSMQ. Is that the still preferred way to do this? Would Reentrant Concurrency mode be a more "modern" option?
There are a lot of alternative queuing systems. Since you have a SQL Server in place I would recommend using MSMQ. In this combination you can use TransactionScope out-of-the-box to handle transcations spanning your DB and the Queuing system.
From my own experience, MSMQ is a proven and stable technology stack.
I'm currently looking to use Microsoft Sync Framework (2.1) to sync clients (running SQL Server Express) with a cloud based central data store, using WCF for all communications.
The central data store is a SQL database, with a scalable number of processing nodes connected to it, each with an instance of my WCF service to process sync calls.
There could be a large amount of data transferred from the server to the clients when syncing, so I think batching is necessary to avoid out of memory issues, better handle unreliable connections, etc. My problem is that the N-tier examples I've seen seem to require an instancing mode of PerSession on the WCF service end, and batch files are stored to a location on disk, which isn't an option as there is no guarantee subsequent calls will go to the same processing node, so my WCF services are all set to PerCall instancing.
What is the best way for me to tackle this batching problem? Is there a way to store the batches on a central data store (say my server database) or is there an alternative to batching to reduce the size of the dataset to 'bite sized' transfers that will be more robust?
the batching in Sync Framework is just for the transmission of the changes, not the application of the changes. so if you have a sync session whose changes are batch into 10 batches, a single batch is not applied individually. rather, the entire 10 batches is applied as one. internally, the batches are actually byte arrays that are reconstructed to a dataset. so you can't have part of the batch in one node and the others on other nodes.
not sure if it helps, but the Windows Azure Sync Service sample may offer you some patterns on how to go about storing the batch file and the writing a similar service and handle the batching.
have a look at Walkthrough of Windows Azure Sync Service Sample
I'm working on a real-time application and building it on Azure.
The idea is that every user reports something about himself and all the other users should see it immediately (they poll the service every seconds or so for new info)
My approach for now was using a Web Role for a WCF REST Service where I'm doing all the writing to the DB (SQL Azure) without a Worker Role so that it will be written immediately.
I've come think that maybe using a Worker Role and a Queue to do the writing might be much more scalable, but might interfere with the real-time side of the service. (The worker role might not take the job immediately from the queue)
Is it true? How should I go about this issue?
Thanks
While it's true that the queue will add a bit of latency, you'll be able to scale out the number of Worker Role instances to handle the sheer volume of messages.
You can also optimize queue-reading by getting more than one message at a time. Since a single queue has a scalability target of 500 TPS, this lets you go well beyond 500 messages per second on reads.
You might look into a Cache for buffering the latest user updates, so when polling occurs, your service reads from cache instead of SQL Azure. That might help as the volume of information increases.
You could have a look at SignalR, it does not support farm scenarios out-of-the-box, but should be able to work with the use of either internal endpoint calls to update every instance, using the Azure Service Bus, or using the AppFabric Cache. This way you get a Push scenario rather than a Pull scenario, thus you don't have to poll your endpoints for potential updates.