Introduction
I decide to share my 10 years experience in multimedia applications development here. I apologize for not huge description as some things for me very trivial and I not willing for describe simple things which can be found over web. I mark this tutorial for intermediate. Let’s roll…
What for this tutorial
This is the standalone application for rendering video over Direct3D in .NET. Application can be used as a start point for implementation rendering your own scenes with the one of couple video sources in it.
What you should know before start
I suggest you to read about following topics over web or on MSDN.
- DirectShow – media playback.
- What is the VMR9 – what is it and usage.
- Interop marshaling and PInvoke.
- Briefly Direct3D 9 and scene presenting.
- Multithreading.
- COM
Application overview
Demo application shows how to perform video playback using DirectShow. It uses the VMR9 rendering filter which available on windows XP and higher systems. VMR9 allows provide your own engine for surface allocation and/or presenting that surfaces on the screen. In sample we use our own stuff for both features allocation and presenting this means that the VMR9 will be used in “Renderless” mode. Presenting surfaces we will do via Direct3D9 and enough for our application will be SlimDX (managed library for Direct9D), but you can use any other libraries or even create managed wrapper for required interfaces. Next figure display the video data flow.
Application makes the custom allocator and provides it to the VMR9 during initialization. On playback VMR9 query for allocated surfaces and perform updating it with the video data after it marks surface ready for display and we send that surface to presenter, and it display them in the application.
My class library
Implementation does not use the DirectShow.NET library, and uses the direct marshaling of DirectShow interfaces. I have couple of classes which helps you to use DirectShow in your application, but you can use DirectShow.NET library in your application. Here I briefly describe my common classes which are used here:
COMHelper – base class with definitions of common types and common values such as HRESULTS values windows API helper macros and functions for trace.
BOOL – class representing of Boolean value.
HRESULT – helper class for COM HRESULT’s values.
FOURCC – Four Character Code implementation.
DSObject<T> - template class for COM objects – here used for filters pins and other objects in DirectShow.
DSPin – helper class for IPin interface.
DSFilter – helper class for IBaseFilter interface.
DSFilterGraphBase – base class for implementing DirectShow filter graph.
DSFilePlayback – base class for media file playback.
More information regarding each class you can find in source code reviewing.
Implementing scene presenting
As I mentioned earlier, I use SlimDX library for Direct3D rendering. So we need to add this library as reference and need to include using aliases:
using SlimDX.Direct3D9;
using SlimDX;
Initialization of the scene class with creating Direct3D device with specified control there we will do presenting incoming video and store main RenderTarget as a class variable:
public Scene(Control _control)
: this(_control, null)
{
}
public Scene(Control _control, Device _device)
{
m_Control = _control;
m_Device = _device;
if (m_Device == null)
{
Direct3DEx _d3d = new Direct3DEx();
DisplayMode _mode = _d3d.GetAdapterDisplayMode(0);
PresentParameters _parameters = new PresentParameters();
_parameters.BackBufferFormat = _mode.Format;
_parameters.BackBufferCount = 1;
_parameters.BackBufferWidth = m_Control.Width;
_parameters.BackBufferHeight = m_Control.Height;
_parameters.Multisample = MultisampleType.None;
_parameters.SwapEffect = SwapEffect.Discard;
_parameters.PresentationInterval = PresentInterval.Default;
_parameters.Windowed = true;
_parameters.DeviceWindowHandle = m_Control.Handle;
_parameters.PresentFlags = PresentFlags.DeviceClip | PresentFlags.Video;
m_Device = new DeviceEx(_d3d, 0, DeviceType.Hardware, m_Control.Handle,
CreateFlags.Multithreaded | CreateFlags.HardwareVertexProcessing, _parameters);
}
m_RenderTarget = m_Device.GetRenderTarget(0);
}
Now it’s time for scene presenting code. Note this code simplify copy surface data into backbuffer surface as I decide not implement fully creating the scene, modern GPU allows to rendering this way, but for old GPU you should make vertex declaration and rendering texture, how to do that proper described in DirectX documentation.
public void OnSurfaceReady(ref Surface _surface)
{
lock (m_csLock)
{
m_Device.SetRenderTarget(0, m_RenderTarget);
m_Device.Clear(ClearFlags.Target, Color.Blue, 1.0f, 0);
m_Device.BeginScene();
Surface _backbufer = m_Device.GetBackBuffer(0, 0);
m_Device.StretchRectangle(_surface, _backbufer, TextureFilter.Linear);
m_Device.EndScene();
m_Device.Present();
}
}
Code lock(m_csLock) is used to multithreaded presenting device access.That’s all for implementing scene initialization and presenting the video, hope you not close your browser so far J
Implementing VMR9 Allocator
Hardest and major part of this application is here.
Filter Graph
The filter graph we are building inside application is particular playback graph but instead of default video renderer we insert VMR9 Renderer filter. So graph will be looks:
Events
First step is delegating the event which will be occurred once surface will be ready:
public delegate void SurfaceReadyHandler(ref Surface _surface);
And event variable in class:
public event SurfaceReadyHandler OnSurfaceReady;
This handler will be connected to the scene method which described earlier.
Initialization
I not describe the DirectShow graph building entirely as there are a lot of articles here and on other resources how to do so, I’ll just my classes for it. VMR9 filter declaration:
[Guid("51b4abf3-748f-4e3b-a276-c828330e926a")]
public class VMR9Renderer : DSFilter
{
public VMR9Renderer()
: base()
{
}
}
Our Allocator class declaration will looks as following:
public class DSFilePlaybackVMR9 : DSFilePlayback, IVMRSurfaceAllocator9, IVMRImagePresenter9
VMR9 class is derived from base graph builder class which handles all basic stuff for file playback over DirectShow we just need to modify initialization method a little:
protected override HRESULT OnInitInterfaces()
{
m_Renderer = new VMR9Renderer();
m_Renderer.FilterGraph = m_GraphBuilder;
IVMRFilterConfig9 _config = (IVMRFilterConfig9)m_Renderer.QueryInterface(typeof(IVMRFilterConfig9).GUID);
HRESULT hr;
if (_config != null)
{
hr = (HRESULT)_config.SetRenderingMode(VMR9Mode.Renderless);
hr.Assert();
hr = (HRESULT)_config.SetNumberOfStreams(5);
hr.Assert();
}
IVMRSurfaceAllocatorNotify9 _notify = (IVMRSurfaceAllocatorNotify9)m_Renderer.QueryInterface(typeof(IVMRSurfaceAllocatorNotify9).GUID);
if (_notify != null)
{
hr = (HRESULT)_notify.AdviseSurfaceAllocator(new IntPtr(g_ciUsedID), this);
hr.Assert();
hr = (HRESULT)this.AdviseNotify(_notify);
hr.Assert();
}
hr = base.OnInitInterfaces();
return hr;
}
Here we insert VMR9 renderer filter into FilterGraph, configure it for “Renderless” mode and specify our own allocator presenter (class derived from IVMRSurfaceAllocator9 and IVMRImagePresenter9 interfaces).
Implementing IVMRSurfaceAllocator9
This interface we should have implement for allocating surfaces for VMR9 filter. Filter calls this interface during media type negotiation for allocating surfaces and during playback for query new surface. First look at the variables in presenter class:
protected Device m_Device = null;
protected const int g_ciUsedID = 0x01020304;
protected IVMRSurfaceAllocatorNotify9 m_lpIVMRSurfAllocNotify = null;
protected object m_csLock = new object();
protected List<Surface> m_Surfaces = new List<Surface>();
protected Texture m_PrivateTexture = null;
That’s all that we need to handle our allocator. First method of the interface called by VMR9 filter is the AdviseNotify, which provide allocator notify object. We will use it to specify our Direct3D device handle to VMR9:
public int AdviseNotify(IVMRSurfaceAllocatorNotify9 lpIVMRSurfAllocNotify)
{
lock (m_csLock)
{
m_lpIVMRSurfAllocNotify = lpIVMRSurfAllocNotify;
if (m_lpIVMRSurfAllocNotify != null)
{
IntPtr hMonitor = m_Device.Direct3D.GetAdapterMonitor(
m_Device.CreationParameters.AdapterOrdinal);
HRESULT hr = (HRESULT)m_lpIVMRSurfAllocNotify.SetD3DDevice(m_Device.ComPointer, hMonitor);
hr.Assert();
return hr;
}
}
return NOERROR;
}
Next call is made during filter connection for allocating surfaces is InitializeDevice. In this method we allocate surfaces and put them into the list.
lpAllocInfo.dwFlags |= VMR9SurfaceAllocationFlags.TextureSurface;
m_Surfaces.Clear();
IntPtr[] lplpSurfaces = new IntPtr[lpNumBuffers];
hr = (HRESULT)m_lpIVMRSurfAllocNotify.AllocateSurfaceHelper(ref lpAllocInfo, ref lpNumBuffers, lplpSurfaces);
hr.Assert();
if (hr.Succeeded)
{
for (int i = 0; i < lplpSurfaces.Length; i++)
{
Marshal.AddRef(lplpSurfaces[i]);
Surface _surface = Surface.FromPointer(lplpSurfaces[i]);
m_Surfaces.Add(_surface);
}
}
During playback VMR9 request surfaces for update:
public int GetSurface(IntPtr dwUserID, int SurfaceIndex, int SurfaceFlags, out IntPtr lplpSurface)
{
ASSERT(dwUserID.ToInt32() == g_ciUsedID);
lplpSurface = IntPtr.Zero;
if (SurfaceIndex > m_Surfaces.Count) return E_INVALIDARG;
lock (m_csLock)
{
lplpSurface = m_Surfaces[SurfaceIndex].ComPointer;
Marshal.AddRef(lplpSurface);
}
return NOERROR;
}
In TerminateDevice we just remove all surfaces from list and release them.
Implementing IVMRImagePresenter9
The presentation part of the VMR9 should implement this interface. Major method here is PresentImage, which called by VMR9 once frame ready for display. In this method we just call the event handler described above with specifying that surface:
public int PresentImage(IntPtr dwUserID, ref VMR9PresentationInfo lpPresInfo)
{
if ((object)lpPresInfo == null)
{
return E_POINTER;
}
else
if (lpPresInfo.lpSurf == IntPtr.Zero)
{
return E_POINTER;
}
lock (m_csLock)
{
Surface _source = Surface.FromPointer(lpPresInfo.lpSurf);
if (_source != null && OnSurfaceReady != null)
{
if (m_PrivateTexture != null)
{
Surface _target = m_PrivateTexture.GetSurfaceLevel(0);
m_Device.StretchRectangle(_source, _target, TextureFilter.None);
OnSurfaceReady(ref _target);
}
else
{
OnSurfaceReady(ref _source);
}
}
}
return NOERROR;
}
Shutdown
We need to release our objects which we are created during FilterGraph initialization:
protected override HRESULT OnCloseInterfaces()
{
if (m_Renderer)
{
m_Renderer.Dispose();
m_Renderer = null;
}
if (m_lpIVMRSurfAllocNotify != null)
{
Marshal.ReleaseComObject(m_lpIVMRSurfAllocNotify);
m_lpIVMRSurfAllocNotify = null;
}
return base.OnCloseInterfaces();
}
Main Application
Implementation of main form is very easy, most interesting methods I describe here. Variable declaration for scene and playback:
private Scene m_Scene = null;
private DSFilePlayback m_Playback = null;
Creating scene object:
m_Scene = new Scene(this.pbView);
Here pbView control on which we’ll do presenting the video. Starting playback code:
m_Playback = new DSFilePlaybackVMR9(m_Scene.Direct3DDevice);
m_Playback.OnPlaybackStop += new EventHandler(btnStart_Click);
((DSFilePlaybackVMR9)m_Playback).OnSurfaceReady += new VMR9.SurfaceReadyHandler(m_Scene.OnSurfaceReady);
m_Playback.FileName = this.tbFileName.Text;
if (m_Playback.Start().Succeeded)
{
btnStart.Text = "Stop";
btnBrowse.Enabled = false;
}
In this code we create VMR9 rendering graph with specified Scene device. After we provide event handler for surface ready notify, and starting playback. Stopping code is fairly simple – just Dispose the playback:
m_Playback.Dispose();
m_Playback = null;
Comments and notes
Any comments are welcome, hope I'll find time to post couple other interesting code.
History
10.07.2012 - initial version.
