u-sql job is very slow, when i add a .NET call - azure-data-lake

Question

The code performs very fast over 2000 small files (~10-50 Kb) ~ 1 min. Parallelizm = 5.
#arenaData =
EXTRACT col1, col2, col3
FROM #in
USING Extractors.Tsv(quoting : true, skipFirstNRows : 1, nullEscape : "\\N", encoding:Encoding.UTF8);
#res =
SELECT col1, col2, col3
FROM #arenaData;
OUTPUT #res
TO #out
USING Outputters.Csv();
But if i change the code like this it takes ~ 1 hour
#arenaData =
EXTRACT col1, col2, col3
FROM #in
USING Extractors.Tsv(quoting : true, skipFirstNRows : 1, nullEscape : "\\N", encoding:Encoding.UTF8);
#res =
SELECT
col1.ToUniversalTime().ToString("yyyy-MM-dd HH:mm:ss", CultureInfo.InvariantCulture) AS col1_converted,
, col2, col3
FROM #arenaData;
OUTPUT #res
TO #out
USING Outputters.Csv();
Why the .NET call so slow? I need to convert the date format in the source CSV files to "yyyy-MM-dd HH:mm:ss"? How can i do that effectively?

Answer 1

Great to hear you are getting the better performance now!
Your job runs on over 2800 very small files using an expression that is being executed in managed code and not translated into C++ as some of the more common C# expressions in U-SQL are.
This leads to the following problem:
You start your job with a certain number of AUs. Each AU then starts a YARN container to execute part of your job. This means that the container needs to be initialized cleanly which takes some time (you can see it in the Vertex Execution View as creation time). Now this takes a bit of time, that is not that much overhead if your vertex does some large amount of processing. Unfortunately in your case, the processing is very quick on a small file so there is a large overhead.
If the vertex only executes system generated code that we codegen into C++ code, then we can reuse containers without the re-initialization time. Unfortunately, we cannot reuse general user-code that gets executed with the managed runtime due to potential artifacts being left behind. So in that case we need to re-initialize the containers which will take time (over 2800 times).
Now based on your feedback, we are improving our reinitialization logic (we can still reinitialize if you do not do anything fancy with inline C# expressions). Also, it will get better once we can process several small files inside a single vertex instead of one file per vertex.
Workarounds for you is to increase the sizes of your files and to possibly avoid custom code (not always possible of course) to be in too many of the vertices.