What is the Flow Run Size Limit of a notebook activity in Azure Synapse? - azure-synapse

Question

I created a large spark notebook and ran it successfully in Azure Synapse. Then I created a new pipeline with a new notebook activity pointing to the existing spark notebook. I triggered it and it failed with the error message:
ErrorCode=FlowRunSizeLimitExceeded, ErrorMessage=Triggering the pipeline failed
due to large run size. This could happen when a run has a large number of
activities or large inputs used in some of the activities, including parameters.
There is only one activity in that pipeline; so, it can't be the number of activities being exceeded. I googled flow run size limit on activity and there was no result. What is the flow run size limit on the notebook activity?
Here is the information:
Filename
Blob Size
UID_ISO_FIPS_LookUp_Table.csv
396 KiB
05-11-2021.csv
630 KiB
https://ghoapi.azureedge.net/api
476 KiB
Type
Size
Cell Total
Cluster Size
.ipynb notebook
668,522 bytes
43 cells
Small (4 vCores / 32 GB) - 3 to 3 nodes
Here is the error message after triggering the pipeline
Here is the sample code in the notebook. The purpose is to join three files into a single file with a single table. Some processing of csv files are filtering, selecting columns, renaming columns, and aggregating values.
Could someone explain why the error message occurred?

Answer 1

I was able to import that .csv with the following Python code on a small Spark pool:
%%pyspark
df = spark.read.load('abfss://someContainer#somestorageAccount.dfs.core.windows.net/raw/csv/05-11-2021.csv', format='csv'
, header = True
)
display(df.limit(10))
df.createOrReplaceTempView("tmp")
Saving it as a temp view allows you to write some conventional SQL to query the dataframe, eg
%%sql
SELECT SUM(deaths) xsum
FROM tmp