We have a few SQL Azure instances which have a bunch of databases (like one for each department in an org) . We want to do stream processing on the data as and when there are updates to data in these individual databases.
From a scale perspective, we are looking at 10K events/day across all the databases, with a possible SLA of a few seconds to stream process an event.
We want to push data to Azure Event Hubs. Any existing Azure product offering can help here?
Just some suggestions for feasible solutions.
If you are looking for a solution for Azure services, then logic app should be able to meet your needs, this is the official doc of sql server trigger in logic app:
https://learn.microsoft.com/en-us/azure/connectors/connectors-create-api-sqlazure#add-a-sql-trigger
https://learn.microsoft.com/en-us/connectors/sql/#when-an-item-is-modified-(v2)
Or, if you can send web request from sql server when items changed like this:
https://www.botreetechnologies.com/blog/how-to-fire-a-web-request-from-microsoft-sql-server
Then you can send it to a endpoint like azure function, and use the event hub output binding, or just use the event hub sdk to send to event hub.
Related
We are using azure iot hub and azure iot edge devices for multiple of our customers. The devices are sending their telemetry data using iot hub telemetry messages. In the iot hub we use message routing -> enrich messages to add the customer id to the messages application properties. This works pretty good if we forward the messages to service bus topics and consume them using azure functions as the application properties are easily accessable there.
Instead of using Azure functions, we now want to store all the telemetry data directly in azure data explorer databases and we want to split the messages to one dedicated database per customer (the name of the database is the customer-id).
So far I could not figure out how to access the application properties from iot hub when importing data to the data explorer? I am not even sure that it is possible, but i really hope it is.
Unfortunately automatically routing messages to different databases is not currently supported, please add a request to Azure Data Explorer user voice. For now, you can do the message routing to a dedicated event hub per customer (i.e. per database) and create one Azure Data Explorer data connection per database.
Please note that within a data connection you can dynamically route the messages to different tables by adding the table information to the event properties, see sample here
We are wanting to use Azure servers to run our Power Apps applications, however we have local SQL servers which contains our data warehouse we want only certain tables to be on Azure and want to create data feeds between the two with information going from one to the other.
Does anyone have any insight into how I can achieve this?
I have googled but there doesn't appear to be a wealth of information on this topic.
It depends on how fast after a change in your source (the on premise SQL Server) you need that change reflected in your Sink (Azure SQL).
If you have some minutes or even only need to update it every day I would suggest a basic Data Factory Pipeline (search on google for data factory upsert). Here it depends on your data on how you can achieve this.
If you need it faster or it is impossible to extract an incremental update from your source you would need to either use triggers and write the changes from one database to the other or get a program that does change data capture that does that.
It looks like you just want to sync the data in some table between local SQL Server and Azure SQL database.
You can use the Azure SQL Data Sync.
Summary：
SQL Data Sync is a service built on Azure SQL Database that lets you synchronize the data you select bi-directionally across multiple SQL databases and SQL Server instances.
With Data Sync, you can keep data synchronized between your on-premises databases and Azure SQL databases to enable hybrid applications.
A Sync Group has the following properties:
The Sync Schema describes which data is being synchronized.
The Sync Direction can be bi-directional or can flow in only one
direction. That is, the Sync Direction can be Hub to Member, or
Member to Hub, or both.
The Sync Interval describes how often synchronization occurs.
The Conflict Resolution Policy is a group level policy, which can be
Hub wins or Member wins.
Next step, you need to learn how to configure the Data Sync. Please reference this Azure document:Tutorial: Set up SQL Data Sync between Azure SQL Database and SQL Server on-premises.
In this tutorial, you learn how to set up Azure SQL Data Sync by creating a sync group that contains both Azure SQL Database and SQL Server instances. The sync group is custom configured and synchronizes on the schedule you set.
Hope this helps.
The most robust solution here is Transactional Replication. You can also use SSIS or Azure Data Factory for copying tables to/from Azure SQL Database. And Azure SQL Data Sync also exists.
We have a multi-tenant, single db application where some customers have expressed the desire to get direct access to their own data.
I have been suggested looking into Azure Data Sync to achieve a setup where each of the customers get their own Azure SQL instance to which we setup a one-way synchronization of their data from the master database.
I managed to find some documentation on this, but one I got around to try it out in a lab setup, it looks like the ability to filter rows in the sync job has been removed in a later iteration of the Azure Data Sync service.
Am I wrong or is that feature really gone? If so, what would be your suggestions to achieve something similar on Azure?
You cannot filter rows using Azure SQL Data Sync. However, you can build a custom solution based on Sync Framework as explained here.
I need to migrate existing SQL broker to Azure SQL.
Can we use Azure Logic apps to perform the same functionality.
Suggestions , Please.
Yes ,you can do it using combination of Azure Web job(read message from source database or application) and Azure Service bus (for queuing and messaging) and Logic App (process data and store in to final database)
please see below diagram for understading
We are at the onset of developing a solution to handle collection and storage of scientific field data.
The solution should handle multiple Thick Windows PC field-clients attached to vehicles (trucks, boats, etc.) connected through cellular-network to a central SQL server.
The clients provide the central server, with data collected from equipment as well as manual input. The clients consume semi-static data from the central server e.g. personnel lists, and predefined data relevant to the specific task.
Connection to the server is erratic and hence the clients should be able to operate fully without connection to the central server for up to 3 hrs.
We are looking at MSMQ and Microsoft Sync Framework as options to handle client/server communication. Any insights you can provide will be much appreciated.
Implement the sync with sync framework over WCF. This will allow you to (a.o.) compress the data with WCf behaviors. And you won't have to expose your sql server to the internets.
http://code.msdn.microsoft.com/Database-Sync-SQL-Server-7e88adab
and http://code.msdn.microsoft.com/Database-SyncSQL-Server-e97d1208
If you can have collisions (update data on multiple clients or both on client and server), implement a command pattern to send data to the server from the clients. Change the data locally on the clients and at the same time create a message to send to the server that does not use sync framework, but can be processed by the server with the same results. This gives you more control and flexibility.
I don't know about msmq. You can have reliable messaging over WCF and as long as the messages you send from the clients are idempotent and the data you send to the clients from the server is considered as the overriding truth, I don't see the need for msmq.
If you can use sql express on the clients, I very much prefer the sync fx 2.0 approach with sql server change tracking, but that's a Microsoft unsupported scenario.
Otherwise, the sync fx 2.1 approach with metadata tables is ok, as long as you don't have more thann, say 50 tables.
If you have more specific questions, I might know more.