DirectShow Hangs on Pause – Solved

Introduction

This article shows how a problem (hangs on play/pause) with a DirectShow-based application was corrected.

Background

There is an application which uses a custom video mixer based upon DirectShow. In deployment, the application would occasionally hang as the user moved a track bar to see different parts of the video.

The application is written in VB.Net with various C# components. The final layer connecting the application to DirectShow is the open-source DirectShow.NET library. The application is part of an established product with a substantial and growing customer base. A workable solution to the problem was required to preserve data compatibility with that customer base. (“Rewrite the application in C++ “ was not considered an economically viable option.)

Engineering problems must be solved within some constraints; otherwise anybody could do it…

The video mixer uses a call back on each frame display and raises an event to be handled by the main program thread. This updates the track bar and position indicators. The raised event runs in a worker thread, which cannot do very much in the main program. The main program uses Windows Forms which is not thread-safe.

VB.Net provides the “INVOKE” statement to handle this sort of problem – but it does not always work in this situation. Under rare timing conditions, the program would hang on the INVOKE statement. This would happen when the user manually adjusts the track bar.

"Rare timing conditions" in practice, meant about 20 minutes of video editing. This was not considered acceptable for the product.

Solution

The problem was solved by using the "PostMessage" function with custom messages to indicate video positioning and overlay requirements. A override to the standard "WNDPROC" method intercepts the custom messages in the main thread, and calls the appropriate event handlers.

Code Fragment

'
' A callback from the VMIXER control is usually in a worker thread, which creates an invalid cross-thread response in
' the main form when VMIXER raises an event.
'
' Handling the callback with an Invoke statement can cause a rare deadlock on a VMIXER Pause, Stop, or Play request,
' as the main thread may be waiting for event flag clearance from the worker thread before processing the Invoke.
'
' So we will send a custom window message back to the main window to deal with the events:
'   "OnVideoPositionChange"
'   "OnVideoOverlay"
'
Private Declare Function PostMessage Lib "user32.dll" Alias "PostMessageA" (ByVal hwnd As IntPtr, ByVal wMsg As Integer, ByVal wParam As IntPtr, ByVal lParam As Integer) As Integer

Private Const MSG_FRAME_BEFORE_CHANGE As Integer = &H4005
Private Const MSG_FRAME_AFTER_CHANGE As Integer = &H4006
Private hMainWindow As IntPtr = Me.Handle       ' Cannot call "Me.Handle" inside "NotifyFrameAdvance" because of cross-threading

Private Sub NotifyFrameAdvance(MsgCode As Integer)
    PostMessage(hMainWindow, MsgCode, IntPtr.Zero, IntPtr.Zero)
End Sub

Protected Overrides Sub WndProc(ByRef wMsg As System.Windows.Forms.Message)
    Dim msgID As Integer = wMsg.Msg
    Select Case msgID
        Case MSG_FRAME_BEFORE_CHANGE
            HandleBeforeFrameAdvance()
        Case MSG_FRAME_AFTER_CHANGE
            HandleAfterFrameAdvance()
        Case Else
            'call the BaseClass handler to ensure all other Windows messages get processed)
            MyBase.WndProc(wMsg)
    End Select
End Sub
'
' This handler gets called just before each frame is rendered by the video mixer.
'
' This event will originate from a cross-thread.
'
Private Sub OnVideoPositionChange(ByVal sender As Object, ByVal e As System.EventArgs) Handles VMIXER.OnVideoPositionChange
    '
    '  We cannot do much in a cross-thread call to a Winform module.
    '  Even an INVOKE here may cause a deadlock with the main thread in rare circumstances.
    '
    ' Just notify the main window that we were here and let the main thread handle the processing.
    '
    NotifyFrameAdvance(MSG_FRAME_BEFORE_CHANGE)
End Sub
'
' This handler gets called just after each frame is rendered by the video mixer.
' It is supposed to overlay the graphics on the frame if that viewing tab is active.
'
' This event will originate from a cross-thread.
'
Private Sub OnVideoOverlay(ByVal sender As Object, ByVal e As System.EventArgs) Handles VMIXER.OnVideoOverlay
    '
    ' Get out quickly if we are not doing overlays
    '
    If CurrentTab <> Tabs.VideoOverlay Then
        Exit Sub
    End If
    '
    ' Tell main window thread that we were here.
    '
    ' Remember, we are running in a worker thread here and cannot do much else safely.
    '
    NotifyFrameAdvance(MSG_FRAME_AFTER_CHANGE)
End Sub

Points of Interest

It appears that the main thread which calls the video “Play”, “Pause”, and “Stop” methods waits on an event in DirectShow before proceeding. That event must be cleared by a worker thread in DirectShow – the same worker thread that does the call back on each frame display.

The WinForms INVOKE processing appears to require a main thread that is active to receive the signal, and does not complete until the main thread acknowledges the signal. If the main thread calls “Pause” at exactly the time a worker thread is starting the call back, the main thread cannot acknowledge the INVOKE, and the worker thread cannot proceed to clear the event upon which the main thread is waiting. 

The main thread and the worker thread are both waiting for each other to release something. This appears to be the reason for the hang.

Several liters of caffeinated beverages (and many days of tracing thread logs) were required to devise this theory. 

The solution worked.

History

August 2014, Initial publication