I have a stored procedure that I use to log the progress of my ADF executions.
I can capture things like Data Factory Name (#pipeline().DataFactory) and RunId (#pipeline().RunId) and record these against the rows in the log table.
However, what I also want to capture is the error output from the notifications tab when executions fails.
For example
I tried this in the failure constraint (red arrow)
#activity('Execute LandingTbls').output
but the output in the log table from this was (not much help here)
System.Collections.Generic.Dictionary`2[System.String,System.Object]
How can this be done?
Basiclly, you can do like this:
The expression is #activity('Validation1').Error.Message.
(On my side, the activity I want to check error message is Validation1, you can change it to the activity on your side.)
Related
I am experimenting with Data Flows in Azure Data Factory.
I have:
Set up a LinkedService to a SQL Server db. This db only has 2 tables.
The two tables are called "dummy_data_table1" and "dummy_data_table1" and are registered as Datasets
The ADF is copying data from these 2 tables, and in the Data Flow they are called "source1" and "source2"
However, when I select a source, go to Source options, and change Input from Table to Query and enter a simple query, it returns 0 columns (there are 11 columns in dummy_data_table1). I suspect my syntax is wrong, but how should I change it?
Hopefully this screenshot will help.
The problem was not the syntax. The problem was that the data flow could not recognize "dummy_data_table1" because it didn't refer to anything known. To make it work, I had to:
Enable Data Flow Debug (at the top of the page, not visible in my screenshot)
Once that's enabled, I had to click on "import projection" to import the schema of my table
Once this is done, the table name and fields are all automatically recognized and can be referenced to in the query just like one would do in SQL Server.
Source:
https://learn.microsoft.com/en-us/azure/data-factory/data-flow-source#import-schema
I'm testing the data flow on my Azure Data Factory. I created Data Flow with the following details:
Source dataset linked service - from CSV files dataset from Blob storage
Sink linked service - Azure SQL database with pre-created table
My CSV files are quite simple as they contain only 2 columns (PARENT, CHILD). So, my table in SQL DB also have only 2 columns.
For the sink setting of my data flow, I have allowed insert data and leaving other options as default.
I have also mapped the 2 columns for input and output columns as per screenshot.
The pipeline with data flow ran successfully when I checked the result, I could see thqat 5732 rows were processed. Is this the correct way to check? As this is the first time I try this functionality in Azure Data Factory.
But, when I click on Data preview tab, they are all NULL value.
And; when I checked my Azure SQL DB in the table where I tried to insert the data from CSV files from Blob storage with selecting top 1000 rows from this table, I don't see any data.
Could you please let me know what I configured incorrectly on my Data Flow? Thank you very much in advance.
Here is the screenshot of ADF data flow source data, it does see the data on the right side as they are not NULL, but on the left side are all NULLs. I imagine that the right side are the data from the CSV from the source on the blob right? And the left side is the sink destination as the table is empty for now?
And here is the screenshot for the sink inspect input, I think this is correct as it reads the 2 columns correctly (Parent, Child), is it?
After adding Map drifted, to map "Parent" => "parent" and "Child" => "child"
I get this error message after running the pipeline.
When checking on sink data preview, I get this error message. It seems like there is incorrect mapping?
I rename the MapDrifted1 expression to "toString(byName('Parent1))" and Child1 as suggested.
The data flow executed successfully, however I still get NULL data in the sink SQL table.
Can you copy/paste the script behind your data flow design graph? Go to the ADF UI, open the data flow, then click the Script button on top right.
In your Source transformation, click on Data Preview to see the data. Make sure you are seeing your data, not NULLs. Also, look at the Inspect on the INPUT for your Sink, to see if ADF is reading additional columns.
I have an application which generates certain reports while values are entered and generate button is clicked on an application.
I know to which database this application is connecting, but don’t know which stored procedure is getting called. The database is having 100s of stored procedures.
So I need to track which stored procedure is called and what arguments are passed while clicking each button on the application.
Guys, please could you help me to create an extended event session which captures every stored procedure call with the arguments passed - in a specific database?
Thanks
JJ
below is the way to capture stored proc details using extended events
1.Go to management ->Extended Events --> sessions-->Right click and say new session wizard and give a new session name
2.In choose template field,dont choose default template and click next..
3.In next screen you will be presented with below screen,search for batch and select sql batch started and completed events as shown in screenshot.. and click next
Note: Extended events so much info ,if you change the channel to debug,you will be presented with many more events like spill to tempdb,cpu..
4.Next screen presents you with options to choose fields of interest,here I choose text,connection id,client name..
5.in the next screen ,select flters based on your choice,i choose databasename ,you can choose batch text as well for a single proc
6.In this screen,you have the options to choose option of storing data,i choose opion1 since ,I may need data sets collected for long period of time.
7.final screen shows you summary and gives you an option to script out what you have done so far .And also start the event session check box in finalscreen as shown in screen below
Now I ran my stored proc in ssms and when finally when I want to see data,i stopped event session .
through File ->new >merge extended events option ,I have the option to choose multiple data sets and it will be shown llike below
I can see my stored proc and fields of interest as highlighted..
I'm trying to create a data sync using Mule Soft so that Db1 is checked for any updates based on LastModified Date and if so the updates are applied to Db2.
I've got the script to work to a point where when the script is first started, the data is copied from Db1 to Db2. After which the script constantly updating the records in Db2. (Below is my flow Diagram)
I've tried to setup recordVars in the message enricher (in Batch_Step) to see if records exists and route them accordingly in Choice (in Batch_Step1).
I've also enabled water mark in Poll for timestamp but nothing is working to avoid constant updating of inserted records.
Below are screenshot of my configs:
Watermark Setup:
Db1 query:
BatchStep Accept Expression:
Message Enricher:
Choice Setup:
Add LastModifiedDate in the Select statement from Db1 so watermark will able to access the field payload.LastModifiedDate.
Also, what is your query in Db2 batch_step? check it, cause it might always getting results that possibly caused to always have payload.size > 0.
I am using the Browser Tool to create a simple dataset with just 1 table with the following schema:
data:integer,count:integer
I am uploading the data using a comma separated csv file.
When I proceed to create the table I can see the new dataset and table in the left side column and next to Job History I see 1 running.
Nothing happens for a long time, even with a small csv file. When I click on the newly created table I get the error Table Not Found
When I refresh the page everything is gone, the dataset and the table.
This looks like some kind of bug, but as I am new with BigQuery I want to make sure I am not doing anything wrong.
If this is a bug, how can I skip it in order to be able to actually create a dataset with a table?
Any tip in the right direction will be much appreciated
If you look at the job history (in the top left corner), you should be able to see the load job that you ran. If it failed, it will show an error.
My assumption is that you ended up running this yesterday when our load jobs were temporarily backed up. When you run a load job, the UI shows a table placeholder, but the table won't actually exist until the load completes. That is why when you clicked on the table it showed as 'not found' since it hadn't really been created yet. That is also why it didn't show up when you reloaded.
We're in the process of increasing capacity by an order of magnitude, so that should be less likely to happen again.
If you do have jobs that failed that you think should have succeeded, please send the job ID and we can investigate.