Introduction
Silverlight 4 did not have P/Invoke support for calling native functions; Silverlight 5 does (see Pete Brown's blog for more details). Our objectives for this HowTo article is no different from my previous article with almost exactly the same Title. That article dealt with COM support. It is important to go through that article first - most of the steps covered there won't be covered here.
So in short, just as the previous article, this HowTo article tries to demonstrate the power of the MediaElement
and MediaStreamSource
classes with P/Invoke support to call native code to play an AVI video.
******************************************************
Please Note the second download (CMS_S5_plusWriteableBitmap
) includes displaying the same Video Samples using the WriteableBitmap
. Code not explained here.
*******************************************************
MediaElement Only
MediaElement (MediaStreamSource) as well as WriteableBitmap Implementation
Background
As of 1st of September 2011, Silverlight 5 RC has been made available for download (see Pete's blog - link above). Silverlight 5 comes with the Platform Invoke support for calling native code (among many other great features).
To recap from the previous article, we need to first derive our custom class from System.Windows.Media.MediaStreamSource
. This will require us to override a number of methods. Without going into too much detail, the methods are OpenMediaAsync
, GetSampleAsync
, CloseMedia
, SeekAsync
, GetDiagnosticsAsync
, and SwitchMediaStreamAsync
. I will not dig deep into defining these methods, but the ones we shall use in our example code are:
OpenMediaAsync
: We override this method, and in it, we initialize and report some metadata about the media by calling the ReportOpenMediaCompleted()
method.
GetSampleAsync
: We override this method and retrieve the next requested sample by the MediaElement
. MediaElement
will call this method every time it needs a sample. To report back to MediaElement
that the sample is ready, we call the ReportGetSampleCompleted()
method.
Our main objective in this article is to write a simple Silverlight application that plays back an AVI video. Well, for the video (.avi) to play, you must have the relevant codec on your machine first.
We shall use the following simple steps to achieve our goal:
- Prepare a simple UI with a
MediaElement
control, two button
s, and a checkbox
.
- Write our custom class that is derived from
System.Windows.Media.MediaStreamSource
and override all the required methods.
- Set our custom class as the source stream to the
MediaElement
control in UI, behind-code.
This sample was tested using Silverlight 5 RC?
Step 1
- Create a new Silverlight 5 Project (VB.NET) and give it any name you wish.
- Make sure you set the Project Properties as above enabling the out-of-browser and ensuring "Require elevated trust ..." is checked.
- Open the default created
UserControl
named MainPage.xaml.
- Change the XAML code to resemble that shown below, which includes the
MediaElement
(named mediaPlayer
) and two button
s (named OpenStream
and CloseStream
).
<Grid x:Name="LayoutRoot" Background="Black">
<MediaElement x:Name="mediaPlayer"
AutoPlay="True"
Stretch="UniformToFill"
Margin="31,61,33,30"
Opacity="1" />
<Button Content="Open AVI File" Height="23" Margin="29,12,0,0"
Name="OpenStream" HorizontalAlignment="Left" Width="123"
VerticalAlignment="Top" />
<Button Content="Close AVI File" Height="23" Margin="175,12,0,0"
Name="CloseStream" HorizontalAlignment="Left" Width="123"
VerticalAlignment="Top" />
</Grid>
The MediaElement
control mediaPlayer
will be used to display our video. The button OpenStream
will be used to initialize our custom MediaStreamSource
object and assign it as a media stream to mediaPlayer
. The button CloseStream
will be used to close and stop the stream.
We will come back to the UI behind code and connect the remaining code.
Step 2
Please follow Step 2 of my previous article as it is almost the same as most of the code in this project.
Let us modify the overridden stub and call the method (retrieveSample
) that retrieves a Sample
everytime the mediaElement
calls for one. Of course, this is not the best implementation - you may want to spawn a thread and immediately return and let the thread signal to mediaElement
when it has a Sample
.
Protected Overrides Sub GetSampleAsync
(mediaStreamType As System.Windows.Media.MediaStreamType)
If mediaStreamType = mediaStreamType.Video Then
retrieveSample()
Return
End If
End Sub
Again, as in the previous article, I decided to create a stream
that contains only one sample at a time.
Let us now look at the method retrieveSample()
.
Private Sub retrieveSample()
If initcapture = 0 Then
timeexpended = timeGetTime() - initialtime
Else
frametime = 0
timeexpended = 0
initialtime = timeGetTime()
End If
If initcapture = 0 Then
If timelapse > 1000 Then
frametime = frametime + 1
Else
frametime = (timeexpended / 1000) * (1000 / _speed)
If frametime >= numFrames Then
initializeFrametime()
End If
End If
Else
initcapture = 0
End If
pDIB = AVIStreamGetFrame(pGetFrameObj, frametime)
If pDIB <> 0 Then
Marshal.Copy(pDIB + Marshal.SizeOf(bih), RGB_Sample, 0, bih.biSizeImage)
If Not RGB_Sample Is Nothing Then
For verticalCount As Integer = RGB_Sample.Length - 1 _
To 0 Step _frameWidth * RGBByteCount * -1
For horizontalCount As Integer = 0 To _frameWidth - 1
pixelPos = verticalCount - (_frameWidth * RGBByteCount) + _
(horizontalCount * RGBByteCount) + 1
RGBA_Sample(j) = RGB_Sample(pixelPos)
RGBA_Sample(j + 1) = RGB_Sample(pixelPos + 1)
RGBA_Sample(j + 2) = RGB_Sample(pixelPos + 2)
RGBA_Sample(j + 3) = 255
j += RGBAByteCount
Next
Next
End If
j = 0
End If
Dim _sample As Sample = New Sample()
_sample.sampleBuffer = RGBA_Sample
_sample.sampleTime = DateTime.Now
_offset = 0
_stream.Seek(0, SeekOrigin.Begin)
_stream.Write(_sample.sampleBuffer, 0, _count)
Dim mediaSample As MediaStreamSample = New MediaStreamSample(
Me._videoDesc,
_stream,
_offset,
_count,
_timeStamp,
Me.emptyDictionary)
_timeStamp += TimeSpan.FromSeconds(1 / _speed).Ticks
Me.ReportGetSampleCompleted(mediaSample)
Return
End Sub
The method above is what gets called everytime a Sample
is requested by the mediaElement
. In order to know what Frame to extract from the video stream, we calculate based on the current time since the time the playback started. so, in short, frametime is the frame number in the video.
We use the native function AVIStreamGetFrame
to return a Pointer
to a Packed DIB by passing it the Frames
object and the frame number to extract. If all goes well, we ignore the Bitmap Info Header and start copying the frame Bytes just after the Info Header the size of the frame in Bytes into RGB_Sample
.
Now we have raw Bytes representing our uncompressed Sample that is three bytes per pixel. THe Sample that we shall pass back to the MediaElement stream
is also uncompressed but of type RGBA which is 4 bytes per pixel. We therefore need to convert from RGB to RGBA by adding an extra byte to represent the Alpha channel.
If you directly assign the first byte from RGB_Sample
to the first byte in RGBA_Sample
, the image will turn out to be upside-down - at least which is what I get. To flip the image to its true orientation, we start assigning the last byte from RGB_Sample
to the first byte of RGBA_Sample
, and for every fourth byte of RGBA_Sample
, we set the Alpha channel to 0xFF or 255
.
RGBA_Sample[Pixel 1] = frameBytes[Pixel N]
RGBA_Sample[Pixel 2] = frameBytes[Pixel N-1]
RGBA_Sample[Pixel 3] = frameBytes[Pixel N-2]
RGBA_Sample[Pixel 4] = 255
...
RGBA_Sample[Pixel M - 3] = frameBytes[Pixel 3]
RGBA_Sample[Pixel M - 2] = frameBytes[Pixel 2]
RGBA_Sample[Pixel M - 1] = frameBytes[Pixel 1]
RGBA_Sample[Pixel M] = 255
The above will not be possible without the code for opening the AVI file and getting the memory Pointers to the file itself and the AVIStream
and Stream
Information. This is made possible by using the native functions AVIFileOpen
, AVIFileGetStream
, AVIStreamInfo
, etc.
Public Function startVideo(fname As String) As Boolean
Dim res As Int32 = AVIFileOpen(pAVIFile, fname, 32, 0)
res = AVIFileGetStream(pAVIFile, pAVIStream, streamtypeVIDEO, 0)
firstFrame = AVIStreamStart(pAVIStream)
numFrames = AVIStreamLength(pAVIStream)
res = AVIStreamInfo(pAVIStream, streamInfo, Marshal.SizeOf(streamInfo))
_speed = streamInfo.dwRate
With bih
.biBitCount = 24
.biClrImportant = 0
.biClrUsed = 0
.biCompression = BI_RGB
.biHeight = streamInfo.rcFrame.bottom - streamInfo.rcFrame.top
.biPlanes = 1
.biSize = 40
.biWidth = streamInfo.rcFrame.right - streamInfo.rcFrame.left
.biXPelsPerMeter = 0
.biYPelsPerMeter = 0
.biSizeImage = (((.biWidth * 3) + 3) And &HFFFC) * .biHeight
End With
_frameWidth = bih.biWidth
_frameHeight = bih.biHeight
_count = _frameHeight * _frameWidth * _framePixelSize
_frameStreamSize = _count
numScans = IIf(_frameHeight > _frameWidth, _frameHeight, _frameWidth)
pGetFrameObj = AVIStreamGetFrameOpen(pAVIStream, bih)
If pGetFrameObj = 0 Then Return False
ReDim RGB_Sample(bih.biSizeImage - 1)
ReDim RGBA_Sample(_count - 1)
initializeFrametime()
Return True
End Function
Step 3
Now, let's write some code that hooks up our UI to our custom derived class to complete our objective.
In our MainPage.xaml code-behind, we first instantiate our custom derived MediaStreamSource
class. To bring everything into action, initialize our custom MediaStreamSource
and call its public
method startVideo(_filename)
to open our video and start buffering such that when our MediaElement
requests for the first sample (and subsequent ones), our derived object will be ready to satisfy those requests. Finally, set our custom MediaStreamSource
object as the source of our media to the MediaElement
and voila, our application is ready to render .avi videos.
Partial Public Class MainPage
Inherits UserControl
Dim _mediaSource As MyDerivedMediaStreamSource
Dim mediaOpen As Boolean = False
Dim _filename As String = "Video_File_Full_Path_Here.avi"
Public Sub New()
InitializeComponent()
End Sub
Private Sub OpenStream_Click(sender As System.Object, _
e As System.Windows.RoutedEventArgs) Handles OpenStream.Click
_mediaSource = New MyDerivedMediaStreamSource()
If _mediaSource.startVideo(_filename) Then
mediaOpen = True
mediaPlayer.SetSource(_mediaSource)
End If
End Sub
Private Sub CloseStream_Click(sender As System.Object, _
e As System.Windows.RoutedEventArgs) Handles CloseStream.Click
If mediaOpen Then
mediaPlayer.Stop()
_mediaSource.closeStream()
_mediaSource = Nothing
mediaOpen = False
End If
End Sub
End Class
Please change "Video_File_Full_Path_Here.avi" to your own file. It must be a .avi file and you must have the relevant codec installed on your computer for the media to be decompressed.
I hope this simple HowTo article was helpful to all.