Is there a way to obtain pipeline activity run details for Azure Synapse from Log Analytics? - azure-log-analytics

Question

I have set up a log analytics workspace and added it to the diagnostic settings on my Synapse workspace. However, I am unable to write a query that extracts pipeline activity information such as dataRead, rowsCopied, etc.
I have tried using
dataReadvar = parse_json(Output).dataRead to extract the JSON within SynapseIntegrationActivityRuns but it doesn’t seem to be able to find ‘Output’.

Answer 1

Azure Data Factory and Azure Synapse Analytics have three groupings of activities: data movement activities, data transformation activities, and control activities. An activity can take zero or more input datasets and produce one or more output datasets.
Check for the sample pipeline for how it is defined and its pipeline activities:
{
"name": "PipelineName",
"properties":
{
"description": "pipeline description",
"activities":
[
],
"parameters": {
},
"concurrency": <your max pipeline concurrency>,
"annotations": [
]
}
}
Below is the JSON format for Top level structure for Execution Activities:
{
"name": "Execution Activity Name",
"description": "description",
"type": "<ActivityType>",
"typeProperties":
{
},
"linkedServiceName": "MyLinkedService",
"policy":
{
},
"dependsOn":
{
}
}
Activity Policy:
{
"name": "MyPipelineName",
"properties": {
"activities": [
{
"name": "MyCopyBlobtoSqlActivity",
"type": "Copy",
"typeProperties": {
...
},
"policy": {
"timeout": "00:10:00",
"retry": 1,
"retryIntervalInSeconds": 60,
"secureOutput": true
}
}
],
"parameters": {
...
}
}
}
Here are few MS Docs as Docs1, Docs2 which are related to your scenario which can help.