Real-Time Console Output Redirection

• Download source and demo project - 22.9 Kb

Introduction

There are several articles on CodeProject and on MSDN that deal with the redirection of Console process' input/output using pipes. Unfortunately, you might have realized that most of the time, you will receive the output of the program as a giant block of data only when the child process terminates. This is a big issue because usually, you spawn a console process that will perform a task for you, and during the execution time, you want to get a feedback.

This article provides background information on why this problem happens, and a nice solution, easy to integrate in your existing programs.

Background

(If you're not interested in the background explanations, you might want to skip directly to Here comes the solution or to the conclusion).

So you have a nice (third-party) console program that performs a long task and prints out progression messages? Everything seems to work normally when you run that program from the command prompt... But as soon as you want to encapsulate it in a nice GUI program that will present the results to the user, things get worse, and you don't get those progression messages until the end of the subprocess.

Well, I have a good news and a bad news:

• the good news is: it's not the fault of your GUI program.
• the bad news is: most console programs behave differently when their output is redirected to pipes!

Why is that?

Let's dig in Microsoft C Runtime (CRT) Library....

The printf function has an immediate effect when the program is using a real console, and seems delayed when the program is redirected to a pipe.. So, let's take a look at its source code... You will quickly find out that there is a buffering system around the stdout stream, and in order to have _ftbuf flush that stream (= output immediately the result of your printf), you have to have _stbuf reach the last return(1);. Unfortunately, when stdout is redirected to a pipe, you will discover that the if (!_isatty(_fileno(stream))) return(0); prevents this from happening.

The Microsoft CRT considers that stdout is not a TTY when it's a pipe, and changes the buffering behaviour !

So, now there are two options:

1. You can add a fflush(stdout); after each output instruction. This works, but that requires you to have the source code of the console program and to modify/recompile it
2. You want a generic solution that works with any console executable. Then follow with me a little further...

But before continuing, it's time to state these:

• This analysis is valid for programs compiled with Microsoft C Runtime Library only.
• It might not be valid with other runtime libraries, but if you see the same symptoms, that means there must be a similar buffering system.
• And finally, the vast majority of console programs out there have been developed with the Microsoft C Runtime Library.

A little deeper in Microsoft C Runtime (CRT) Library....

OK, so, what would be nice is that we cheat the CRT into thinking that stdout is still a TTY when it's a pipe.

Looking at the source code of _isatty, we need the FDEV flag on our file, and this only happens if a call to GetFileType returns FILE_TYPE_CHAR.

Doh'! For a pipe, GetFileType returns FILE_TYPE_PIPE. And MSDN tells us that FILE_TYPE_CHAR is only returned for printers and the console...

So we really need a console... Are we stuck?

Here comes the solution!

My solution is to really use a console buffer that the father process will create, share, and monitor while the child process writes into it.

But there are two drawbacks that we have to solve:

• Output console buffers can only be written to, not read.
• The only way to have a console buffer is to either be a console process or to call AllocConsole. But we don't want a console window to appear!

The first one is solved by using ReadConsoleOutputCharacter and other console-specific functions that allow us to read information from the console buffer as if we were reading it on the screen.

The second one will be solved in a very elegant way: Instead of calling AllocConsole from our GUI program and quickly find the window in order to hide it (like some articles suggest), we will create an intermediate stub program that our GUI program will spawn instead of spawning the target program.

This little stub program will be a real console process that will be in charge of monitoring the console buffer and flushing the data read onto its own output stream (that our GUI program will redirect to a pipe)

This solution brings two nice advantages:

• The stub program can be run hidden with the SW_HIDE startup window option, so no console window will be visible.
• If you have already written your GUI program using redirection pipes, you can keep it! The only thing you will have to change is to insert "RTconsole.exe" at the beginning of your CreateProcess command-line.

A look at the RTconsole source code

The arguments to RTconsole.exe will be the original command-line, including the path to the target console program.

We build an inheritable console screen buffer and fill it with zeroes:

SECURITY_ATTRIBUTES sa;
sa.nLength = sizeof(sa);
sa.lpSecurityDescriptor = NULL;
sa.bInheritHandle = TRUE;
HANDLE hConsole = CreateConsoleScreenBuffer(GENERIC_READ|
  GENERIC_WRITE,FILE_SHARE_READ|FILE_SHARE_WRITE,
  &sa,CONSOLE_TEXTMODE_BUFFER,NULL);
// fill screen buffer with zeroes
FillConsoleOutputCharacter(hConsole, '\0', 
               MAXLONG, origin, &dwDummy);
// to be inherited by child process
SetStdHandle(STD_OUTPUT_HANDLE, hConsole);

We could have used the standard console that comes with any console program, but this is cleaner and it avoids mixing the subprocess output and our own output in the same console screen buffer. The zeroes will allow us to differentiate with space character outputs. Please note also that CreateConsoleScreenBuffer is possible only because we are a console application. In a GUI application, this would require calling AllocConsole which would display a console popup window.

Now, we start the target process normally, sharing the same console. (RTconsole.exe itself must be started with SW_HIDE to hide the shared console.)

PROCESS_INFORMATION pi;
STARTUPINFO si;
ZeroMemory(&si, sizeof(STARTUPINFO));
si.cb = sizeof(STARTUPINFO);
// we don't want the "app starting" cursor
si.dwFlags = STARTF_FORCEOFFFEEDBACK;
// all other default options are already good :
// we want subprocess to share the same
// console and to inherit our STD handles
if (!CreateProcess(    NULL, commandLine, NULL, NULL, 
     TRUE, 0, NULL, NULL, &si, &pi))

There seems to be no way to be notified on new characters arriving in the console screen buffer, that's why we now need a monitoring loop:

do    {
    if (WaitForSingleObject(pi.hProcess, 0) != WAIT_TIMEOUT)
        exitNow = true; // exit after this last iteration
    // get screen buffer state
    ...
    // loop until end of subprocess
} while (!exitNow);

We exit that loop when the child process has exited, after doing an additional iteration of the loop, for the very last characters output to be taken in account. Monitoring the child process this way also should solve the problem encountered with blocking ReadFile on 16-bit subprocesses.

In the loop, we monitor if the text cursor has moved since the last monitoring, we read the characters on screen from the last known cursor position up to the current position, we fill back with zeroes the portion we have read, and reset the text cursor to its home position.

GetConsoleScreenBufferInfo(hConsole, &csbi);
int lineWidth = csbi.dwSize.X;

if ((csbi.dwCursorPosition.X == lastpos.X) && 
    (csbi.dwCursorPosition.Y == lastpos.Y))
    Sleep(SLEEP_TIME);
    // text cursor did not move, sleep a while
else
{
    DWORD count = (csbi.dwCursorPosition.Y-lastpos.Y)*
                   lineWidth+csbi.dwCursorPosition.X-lastpos.X;
    // read newly output characters
    // starting from last cursor position
    LPTSTR buffer = (LPTSTR) LocalAlloc(0, count*sizeof(TCHAR));
    ReadConsoleOutputCharacter(hConsole, 
        buffer, count, lastpos, &count);
    // fill screen buffer with zeroes
    FillConsoleOutputCharacter(hConsole, '\0', 
                    count, lastpos, &dwDummy);
    ...
    // scan screen buffer and transmit
    // character to real output handle
    ...
    LocalFree(buffer);
}

Then, we analyze the characters read from the screen buffer, and convert them to lines that are written (flushed) to RTconsole's own original output stream.

Weak points

These have been tested to ensure no problem is happening most of the time, but it's always good to know your weak points

Synchronization issues

There is no atomic way of reading the screen buffer and resetting it for the ongoing incoming data. So while we are reading and clearing the screen buffer, there might have been additional characters written by the child process. That's why I'm switching temporarily to THREAD_PRIORITY_TIME_CRITICAL for a quick check to see if the text cursor has moved since then. Characters are not lost in this case because we only clear the characters we have read

Scrolling screen buffer

If the text cursor has not moved, we reset it back to its home position (0,0) to avoid letting it go down the default 25 lines or so of the screen buffer. Otherwise, the screen would start scrolling, and if it happens, then we might lose some text. Note that you could probably use SetConsoleScreenBufferSize to enlarge the screen buffer if the target console program outputs characters too quickly.

Conclusion

I went a bit into details with this article, but remember, in the end:

All you have to do in your GUI application is to insert "RTconsole.exe" at the beginning of the CreateProcess command-line and read the redirected output pipe as usual (see demo program).