Process with redirected standard input/output behaves differently depending on size of input - process

Question

I have an odd behavior about running processes from F#. My basic problem was to run Graphviz's dot.exe with a generated graph, and visualize it.
When I restricted the graph to be small, everything worked fine. But with bigger graphs, it hanged on a specific line. I'm curious why is it happening, so maybe I can fix my issue. Tho I created an MVCE for the purpose.
I have 2 console programs. One is the simulator of dot.exe, where I would showel the input, and expect .jpg from. This version just streams 10 times the input to the output in blocks:
// Learn more about F# at http://fsharp.org
// See the 'F# Tutorial' project for more help.
open System
open System.IO
[<EntryPoint>]
let main argv =
let bufferSize = 4096
let mutable buffer : char [] = Array.zeroCreate bufferSize
// it repeats in 4096 blocks, but it doesn't matter. It's a simulation of outputing 10 times amount output.
while Console.In.ReadBlock(buffer, 0, bufferSize) <> 0 do
for i in 1 .. 10 do
Console.Out.WriteLine(buffer)
0 // return an integer exit code
So I have an .exe named C:\Users\MyUserName\Documents\Visual Studio 2015\Projects\ProcessInputOutputMVCE\EchoExe\bin\Debug\EchoExe.exe Than comes another console project, which uses it:
// Learn more about F# at http://fsharp.org
// See the 'F# Tutorial' project for more help.
open System.IO
[<EntryPoint>]
let main argv =
let si =
new System.Diagnostics.ProcessStartInfo(#"C:\Users\MyUserName\Documents\Visual Studio 2015\Projects\ProcessInputOutputMVCE\EchoExe\bin\Debug\EchoExe.exe", "",
// from Fake's Process handling
#if FX_WINDOWSTLE
WindowStyle = ProcessWindowStyle.Hidden,
#else
CreateNoWindow = true,
#endif
UseShellExecute = false,
RedirectStandardOutput = true,
RedirectStandardError = true,
RedirectStandardInput = true)
use p = new System.Diagnostics.Process()
p.StartInfo <- si
if p.Start() then
let input =
Seq.replicate 3000 "aaaa"
|> String.concat "\n"
p.StandardInput.Write input
// hangs on Flush()
p.StandardInput.Flush()
p.StandardInput.Close()
use outTxt = File.Create "out.txt"
p.StandardOutput.BaseStream.CopyTo outTxt
// double WaitForExit because of https://msdn.microsoft.com/en-us/library/system.diagnostics.process.standardoutput(v=vs.110).aspx
// saying first WaitForExit() waits for StandardOutput. Next is needed for the whole process.
p.WaitForExit()
p.WaitForExit()
0 // return an integer exit code
Which hangs on p.StandardInput.Flush(). Except if I change the volume of input to Seq.replicate 300 "aaaa". Why is it working differently?
Process.StandardOutput's reference states that reading from StandardOutput and child process writing to that stream in the same time might cause deadlock. Who is a child process in that case? Is it my p.StandardInput.Write input?
Other possible deadlock would be
read all text from both the standard output and standard error streams.
But I'm not reading the error stream. Anyway it suggests to handle the input/output with async so I have a following version:
// same as before
...
if p.Start() then
let rec writeIndefinitely (rows: string list) =
async {
if rows.Length = 0 then
()
else
do! Async.AwaitTask (p.StandardInput.WriteLineAsync rows.Head)
p.StandardInput.Flush()
do! writeIndefinitely rows.Tail
}
let inputTaskContinuation =
Seq.replicate 3000 "aaaa"
|> Seq.toList
|> writeIndefinitely
Async.Start (async {
do! inputTaskContinuation
p.StandardInput.Close()
}
)
let bufferSize = 4096
let mutable buffer : char array = Array.zeroCreate bufferSize
use outTxt = File.CreateText "out.txt"
let rec readIndefinitely() =
async {
let! readBytes = Async.AwaitTask (p.StandardOutput.ReadAsync (buffer, 0, bufferSize))
if readBytes <> 0 then
outTxt.Write (buffer, 0, buffer.Length)
outTxt.Flush()
do! readIndefinitely()
}
Async.Start (async {
do! readIndefinitely()
p.StandardOutput.Close()
})
// with that it throws "Cannot mix synchronous and asynchronous operation on process stream." on let! readBytes = Async.AwaitTask (p.StandardOutput.ReadAsync (buffer, 0, bufferSize))
//p.BeginOutputReadLine()
p.WaitForExit()
// using dot.exe, it hangs on the second WaitForExit()
p.WaitForExit()
Which doesn't hang, and writes out.txt. Except in the real code using dot.exe. It's as async as it gets. Why does it throw exception for p.BeginOutputReadLine()?
After some experimenting, the whole async out can stay p.StandardOutput.BaseStream.CopyTo outTxt where outTxt is File.Create not File.CreateText. Only the async input behaves correct against the synchronous input handling. Which is weird.
To sum it up. If I have asynchronous input handling, it works fine (except dot.exe, but if I figure it out, maybe I can fix that too), if it have synchronous input handling, it's depending on the size of the input. (300 works, 3000 doesn't) Why is that?
Update
Since I really not needed to redirect the standard error, I have removed RedirectStandardError = true. That solved the mysterious dot.exe problem.

Answer 1

I think that the deadlock here is following:
Host process writes too much data to the input buffer.
Child process reads from the buffer and writes to output.
Host process does not read from the buffer (it happens after sending all the data). When the output buffer of the child process is filled up, it blocks on writing and stops reading from the input. Two processes are now in the deadlock state.
It is not necessary to use completely asynchronous code. What I think would work is writing in chunks to dot's stdin and reading to the end of the dot's stdout before writing again.