Play AVI files in Silverlight 5 using MediaElement and MediaStreamSource - P/Invoke

• Download source - 78.15 KB (MediaElement Only)
• Download CMSS_S5_plusWriteableBitmap - 86.79 KB (MediaElement plus WriteableBitmap)

Introduction

Silverlight 4 did not have P/Invoke support for calling native functions; Silverlight 5 does (see Pete Brown's blog for more details). Our objectives for this HowTo article is no different from my previous article with almost exactly the same Title. That article dealt with COM support. It is important to go through that article first - most of the steps covered there won't be covered here.

So in short, just as the previous article, this HowTo article tries to demonstrate the power of the MediaElement and MediaStreamSource classes with P/Invoke support to call native code to play an AVI video.

******************************************************
Please Note the second download (CMS_S5_plusWriteableBitmap) includes displaying the same Video Samples using the WriteableBitmap. Code not explained here.
*******************************************************

MediaElement Only

MediaElement (MediaStreamSource) as well as WriteableBitmap Implementation

Background

As of 1^st of September 2011, Silverlight 5 RC has been made available for download (see Pete's blog - link above). Silverlight 5 comes with the Platform Invoke support for calling native code (among many other great features).

To recap from the previous article, we need to first derive our custom class from System.Windows.Media.MediaStreamSource. This will require us to override a number of methods. Without going into too much detail, the methods are OpenMediaAsync, GetSampleAsync, CloseMedia, SeekAsync, GetDiagnosticsAsync, and SwitchMediaStreamAsync. I will not dig deep into defining these methods, but the ones we shall use in our example code are:

• OpenMediaAsync: We override this method, and in it, we initialize and report some metadata about the media by calling the ReportOpenMediaCompleted() method.
• GetSampleAsync: We override this method and retrieve the next requested sample by the MediaElement. MediaElement will call this method every time it needs a sample. To report back to MediaElement that the sample is ready, we call the ReportGetSampleCompleted() method.

Our main objective in this article is to write a simple Silverlight application that plays back an AVI video. Well, for the video (.avi) to play, you must have the relevant codec on your machine first.

We shall use the following simple steps to achieve our goal:

• Prepare a simple UI with a MediaElement control, two buttons, and a checkbox.
• Write our custom class that is derived from System.Windows.Media.MediaStreamSource and override all the required methods.
• Set our custom class as the source stream to the MediaElement control in UI, behind-code.

This sample was tested using Silverlight 5 RC?

Step 1

• Create a new Silverlight 5 Project (VB.NET) and give it any name you wish.
• Make sure you set the Project Properties as above enabling the out-of-browser and ensuring "Require elevated trust ..." is checked.
• Open the default created UserControl named MainPage.xaml.
• Change the XAML code to resemble that shown below, which includes the MediaElement (named mediaPlayer) and two buttons (named OpenStream and CloseStream).

<Grid x:Name="LayoutRoot" Background="Black">
        <MediaElement x:Name="mediaPlayer" 
                      AutoPlay="True"
                      Stretch="UniformToFill"
                      Margin="31,61,33,30"
                      Opacity="1" />
        <Button Content="Open AVI File" Height="23" Margin="29,12,0,0" 
	Name="OpenStream" HorizontalAlignment="Left" Width="123" 
	VerticalAlignment="Top" />
        <Button Content="Close AVI File" Height="23" Margin="175,12,0,0" 
	Name="CloseStream" HorizontalAlignment="Left" Width="123" 
	VerticalAlignment="Top" />
</Grid>        

The MediaElement control mediaPlayer will be used to display our video. The button OpenStream will be used to initialize our custom MediaStreamSource object and assign it as a media stream to mediaPlayer. The button CloseStream will be used to close and stop the stream.

We will come back to the UI behind code and connect the remaining code.

Step 2

Please follow Step 2 of my previous article as it is almost the same as most of the code in this project.

Let us modify the overridden stub and call the method (retrieveSample) that retrieves a Sample everytime the mediaElement calls for one. Of course, this is not the best implementation - you may want to spawn a thread and immediately return and let the thread signal to mediaElement when it has a Sample.

Protected Overrides Sub GetSampleAsync
	(mediaStreamType As System.Windows.Media.MediaStreamType)

        If mediaStreamType = mediaStreamType.Video Then

            retrieveSample()

            Return

        End If

End Sub    

Again, as in the previous article, I decided to create a stream that contains only one sample at a time.

Let us now look at the method retrieveSample().

Private Sub retrieveSample()

        '---------------------------------------------------------------
        ' I CANNOT REMEMBER WHERE I GOT THIS CALCULATION FROM ONLINE
        ' it basically calculates the frame number based on time passed
        ' since start of play
        '-------------------------**********----------------------------
        If initcapture = 0 Then
            timeexpended = timeGetTime() - initialtime
        Else
            frametime = 0
            timeexpended = 0

            initialtime = timeGetTime()
        End If

        If initcapture = 0 Then
            If timelapse > 1000 Then
                frametime = frametime + 1
            Else
                frametime = (timeexpended / 1000) * (1000 / _speed) 'timelapse)

                If frametime >= numFrames Then
                    initializeFrametime()
                End If
            End If
        Else
            initcapture = 0
        End If
        '-------------------------**********----------------------------
        '
        '---------------------------------------------------------------

        ' Get the requested Frame from the Stream - frame number "frametime" provided.
        ' Return a Pointer to the DIB
        pDIB = AVIStreamGetFrame(pGetFrameObj, frametime)


        If pDIB <> 0 Then

            'Copy the Frame bits into RGB_Sample
            'Call CopyMemory(RGB_Sample(0), pDIB + Marshal.SizeOf(bih), bih.biSizeImage)
            Marshal.Copy(pDIB + Marshal.SizeOf(bih), RGB_Sample, 0, bih.biSizeImage)

            If Not RGB_Sample Is Nothing Then

                For verticalCount As Integer = RGB_Sample.Length - 1 _
			To 0 Step _frameWidth * RGBByteCount * -1

                    For horizontalCount As Integer = 0 To _frameWidth - 1

                        ' Calculate the next pixel position from the original Sample
                        ' based on the outer loop, it is calculated from bottom-right
                        pixelPos = verticalCount - (_frameWidth * RGBByteCount) + _
				(horizontalCount * RGBByteCount) + 1

                        RGBA_Sample(j) = RGB_Sample(pixelPos)
                        RGBA_Sample(j + 1) = RGB_Sample(pixelPos + 1)
                        RGBA_Sample(j + 2) = RGB_Sample(pixelPos + 2)

                        ' Assign 1 byte for the Alpha Channel
                        RGBA_Sample(j + 3) = 255 '&FF

                        'jump 4 bytes for the RGBA byte counter
                        j += RGBAByteCount

                    Next
                Next

            End If

            j = 0

        End If

        ' Instantiate a Sample
        ' The Sample has two members (Time & Buffer)           
        Dim _sample As Sample = New Sample()
        _sample.sampleBuffer = RGBA_Sample
        _sample.sampleTime = DateTime.Now


        ' We always start at the beginning of the stream
        ' this is because we always reset the stream with one sample at a time
        ' if you decide to add more than one sample into the stream then you
        ' can modify the logic to increment this offset by the size of the sample
        ' everytime there is a call to return a sample
        _offset = 0
        _stream.Seek(0, SeekOrigin.Begin)

        ' write the retrieved Sample into the stream
        ' remember our stream is just one Sample
        _stream.Write(_sample.sampleBuffer, 0, _count)

        Dim mediaSample As MediaStreamSample = New MediaStreamSample(
                Me._videoDesc,
                _stream,
                _offset,
                _count,
                _timeStamp,
                Me.emptyDictionary)

        _timeStamp += TimeSpan.FromSeconds(1 / _speed).Ticks

        ' report back a successful Sample
        Me.ReportGetSampleCompleted(mediaSample)

        Return

End Sub  

The method above is what gets called everytime a Sample is requested by the mediaElement. In order to know what Frame to extract from the video stream, we calculate based on the current time since the time the playback started. so, in short, frametime is the frame number in the video.

We use the native function AVIStreamGetFrame to return a Pointer to a Packed DIB by passing it the Frames object and the frame number to extract. If all goes well, we ignore the Bitmap Info Header and start copying the frame Bytes just after the Info Header the size of the frame in Bytes into RGB_Sample.

Now we have raw Bytes representing our uncompressed Sample that is three bytes per pixel. THe Sample that we shall pass back to the MediaElement stream is also uncompressed but of type RGBA which is 4 bytes per pixel. We therefore need to convert from RGB to RGBA by adding an extra byte to represent the Alpha channel.

If you directly assign the first byte from RGB_Sample to the first byte in RGBA_Sample, the image will turn out to be upside-down - at least which is what I get. To flip the image to its true orientation, we start assigning the last byte from RGB_Sample to the first byte of RGBA_Sample, and for every fourth byte of RGBA_Sample, we set the Alpha channel to 0xFF or 255.

' assume N number of pixels in the original RGB Sample
' and M pixels in RGBA Sample
RGBA_Sample[Pixel 1] = frameBytes[Pixel N]
RGBA_Sample[Pixel 2] = frameBytes[Pixel N-1]
RGBA_Sample[Pixel 3] = frameBytes[Pixel N-2]
RGBA_Sample[Pixel 4] = 255
...
RGBA_Sample[Pixel M - 3] = frameBytes[Pixel 3]
RGBA_Sample[Pixel M - 2] = frameBytes[Pixel 2]
RGBA_Sample[Pixel M - 1] = frameBytes[Pixel 1]
RGBA_Sample[Pixel M] = 255     

The above will not be possible without the code for opening the AVI file and getting the memory Pointers to the file itself and the AVIStream and Stream Information. This is made possible by using the native functions AVIFileOpen, AVIFileGetStream, AVIStreamInfo, etc.

Public Function startVideo(fname As String) As Boolean

        ' Return a Pointer of the File in Memory assigned to pAVIFile
        Dim res As Int32 = AVIFileOpen(pAVIFile, fname, 32, 0)

        ' Return a Pointer of the Video Stream in Memory assigned to pAVIStream
        res = AVIFileGetStream(pAVIFile, pAVIStream, streamtypeVIDEO, 0)

        ' Return the First Video Frame Number
        firstFrame = AVIStreamStart(pAVIStream)

        ' Return the Total Number of Frames in the Stream
        numFrames = AVIStreamLength(pAVIStream)

        ' Return the Video Stream Information
        res = AVIStreamInfo(pAVIStream, streamInfo, Marshal.SizeOf(streamInfo))

        _speed = streamInfo.dwRate 'fps

        With bih
            .biBitCount = 24
            .biClrImportant = 0
            .biClrUsed = 0
            .biCompression = BI_RGB
            .biHeight = streamInfo.rcFrame.bottom - streamInfo.rcFrame.top
            .biPlanes = 1
            .biSize = 40
            .biWidth = streamInfo.rcFrame.right - streamInfo.rcFrame.left
            .biXPelsPerMeter = 0
            .biYPelsPerMeter = 0
            .biSizeImage = (((.biWidth * 3) + 3) And &HFFFC) * .biHeight
        End With

        _frameWidth = bih.biWidth
        _frameHeight = bih.biHeight

        _count = _frameHeight * _frameWidth * _framePixelSize
        _frameStreamSize = _count

        numScans = IIf(_frameHeight > _frameWidth, _frameHeight, _frameWidth)

        ' Retrieve the Frames Object from the stream byte by supplying 
        ' the Stream and the format we expect
        pGetFrameObj = AVIStreamGetFrameOpen(pAVIStream, bih)

        If pGetFrameObj = 0 Then Return False 'ERROR

        ' Resize our actual Samples in Bytes
        ReDim RGB_Sample(bih.biSizeImage - 1)

        ' Resize our final modified sample in Bytes
        ReDim RGBA_Sample(_count - 1)

        initializeFrametime()

        Return True

End Function    

Step 3

Now, let's write some code that hooks up our UI to our custom derived class to complete our objective.

In our MainPage.xaml code-behind, we first instantiate our custom derived MediaStreamSource class. To bring everything into action, initialize our custom MediaStreamSource and call its public method startVideo(_filename) to open our video and start buffering such that when our MediaElement requests for the first sample (and subsequent ones), our derived object will be ready to satisfy those requests. Finally, set our custom MediaStreamSource object as the source of our media to the MediaElement and voila, our application is ready to render .avi videos.

Partial Public Class MainPage
    Inherits UserControl

    ' Instantiate our derived MediaStreamSource class
    Dim _mediaSource As MyDerivedMediaStreamSource

    '<summary>
    '    Flag to indicate if our media has been opened or not
    '</summary>
    Dim mediaOpen As Boolean = False

    Dim _filename As String = "Video_File_Full_Path_Here.avi"

    'Dim aviObj As cAVIDefs

    Public Sub New()

        InitializeComponent()

    End Sub

    Private Sub OpenStream_Click(sender As System.Object, _
	e As System.Windows.RoutedEventArgs) Handles OpenStream.Click

        ' initialize our media stream object
        _mediaSource = New MyDerivedMediaStreamSource()

        ' check if we succeeded in opening the AVI
        If _mediaSource.startVideo(_filename) Then

            ' set flag to true - media has been opened
            mediaOpen = True

            ' set the source of our media stream to the MediaElement
            mediaPlayer.SetSource(_mediaSource)

        End If

    End Sub

    Private Sub CloseStream_Click(sender As System.Object, _
	e As System.Windows.RoutedEventArgs) Handles CloseStream.Click

        If mediaOpen Then ' check if we still have the media open

            mediaPlayer.Stop()

            _mediaSource.closeStream()


            _mediaSource = Nothing

            mediaOpen = False

        End If
    End Sub

End Class 

Please change "Video_File_Full_Path_Here.avi" to your own file. It must be a .avi file and you must have the relevant codec installed on your computer for the media to be decompressed.

I hope this simple HowTo article was helpful to all.