Lazarus (freepascal) Reading large output from TProcess - process

Question

I' m reading large process output data in Lazarus using the TProcess and the suggestions from this freepascal wiki page.
The wiki page suggests to create a loop to read the process output data like this:
// ... If you want to read output from an external process, this is the code you should adapt for production use.
while True do
begin
MemStream.SetSize(BytesRead + 2024); // make sure we have room
NumBytes := OurProcess.Output.Read((MemStream.Memory + BytesRead)^, READ_BYTES);
if NumBytes > 0
then begin
Inc(BytesRead, NumBytes);
Write('.') //Output progress to screen.
end else
BREAK // Program has finished execution.
end;
// "Then read the MemStream to do your job"
The wiki page also mentions that the calling program should read from the output pipe to prevent it from getting full.
So, how much data makes the output pipe full?
Why we should use a MemStream (TMemoryStream) and not directly read from OurProcess.Output stream (using the bytesAvailable, etc) in the above loop?
I'm reading 80MB of wav data from a process and I have noticed that both MemStream and OurProcess.Output streams have the same amount of data! The memory usage gets doubled. So the suggested method from the wiki cannot be considered as efficient or optimized. Or there is something I'm missing?

Answer 1

Afaik output/input streams are a stream form of a pipe, not memory streams. The values you see are retrieved from the OS handle, not from memory allocated to the FPC app per se.
It is just like you can ask for the .size of a file on disk without reading the whole file.

Answer 2

procedure RunExternalAppInMemo(DosApp:String;AMemo:TMemo);
const READ_BYTES = 2048;
var
aProcess: TProcess; //TProcess is crossplatform is best way
MemStream: TMemoryStream;
NumBytes: LongInt;
BytesRead: LongInt;
Lines: TStringList;
begin
// A temp Memorystream is used to buffer the output
MemStream := TMemoryStream.Create;
Lines :=TStringList.Create;
BytesRead := 0;
aProcess := TProcess.Create(nil);
aProcess.CommandLine := DosApp;
aprocess.ShowWindow := swoHIDE;
AProcess.Options := AProcess.Options + [poUsePipes];
aProcess.Execute;
while aProcess.Running do
begin
// make sure we have room
MemStream.SetSize(BytesRead + READ_BYTES);
// try reading it
NumBytes := aProcess.Output.Read((MemStream.Memory + BytesRead)^, READ_BYTES);
if NumBytes > 0 // All read() calls will block, except the final one.
then Inc(BytesRead, NumBytes)
else
BREAK // Program has finished execution.
end;
MemStream.SetSize(BytesRead);
Lines.LoadFromStream(MemStream);
AMemo.lines.AddStrings(Lines);
aProcess.Free;
Lines.Free;
MemStream.Free;
end;

Answer 3

I was dealing with this problem today, I've modified Georgescu answer, as I wanted Memo to display output stream on the fly
procedure RunExternalAppInMemo(DosApp:String;AMemo:TMemo);
const READ_BYTES = 2048;
var
aProcess: TProcess; //TProcess is crossplatform is best way
NumBytes: LongInt;
Buffer: array of byte;
begin
// set the size of your buffer
SetLength(Buffer,READ_BYTES);
aProcess := TProcess.Create(nil);
aProcess.CommandLine := DosApp;
aprocess.ShowWindow := swoHIDE;
AProcess.Options := AProcess.Options + [poUsePipes];
aProcess.Execute;
while aProcess.Running do
begin
// try reading it
NumBytes := aProcess.Output.Read(Buffer[0], length(buffer)*sizeof(byte)); // I usually do it that way, so I can change Buffer size on if needed
AProcess.Suspend; //I have no experience with pipes, but it seems way I won loose eny output?
if NumBytes > 0 then // All read() calls will block, except the final one.
begin
AMemo.Lines.Add(Pchar(Buffer);
application.ProcessMessages;
AProcess.Resume;
end
else
BREAK; // Program has finished execution.
end;
setlength(Buffer,0);
aProcess.Free;
end;