Azure Data Factory - How to find total count of objects with more than 1 files with same "prefix" in an ADF expression? - azure-data-factory-2

Question

Let's say have bunch of random sample files in a Blob which I want to copy into datalake as .parquet using ADF copy.
abc.1.txt,
abc.2.txt,
abc.3.txt,
def.1.txt,
ghi.1.txt,
xyz.1.txt,
xyz.2.txt
All abc & xyz object files should be merged/appended into their respective single .parquet file and remaining def, ghi as its individual .parquet file in data lake.
Need the output something like:
[
{
name: abc
count: 3
},
name: def
count: 1
},
name: ghi
count: 1
},
name: xyz
count: 2
}
]
Pipeline flow would look something similar:
GetMetadata -->Filter if only 1 file -->Run ForEach file -->Copy activity(without merge)
GetMetadata -->SetVariable-->Filter if >1 file -->Run ForEach file -->Copy (with merge)
However, how do I get the count() of total files with same prefix in the Filter activity ?

Answer 1

A quick thought,
You could get the file details in get meta data activity and push that to a SQL table and do a group by there and return count.
And loop over the result set in foreach and use if condition to check the count.
Here is what I'm doing.
My data lake:
Firstly, get the childitems in getmetadata.
I'm then writing it to a string variable (optional)
From the string variable write to a text file
From the text file to Azure SQL server table.
Use the below query in the script activity
select count(1), fileprefix from( select
substring(name,0,charindex('.',name)-1) fileprefix,type from
[dbo].[temptest] CROSS APPLY OPENJSON(json) WITH(
name varchar(200)
, type varchar(60)
) as my_json_array)a group by fileprefix
My script output:
Thanks.