Invoke FFmpeg to export frames

Introduction

I've already published the article "Another FFmpeg.exe C# Wrapper" using FFmpeg.exe via commandline parameters to extract frame images. But that method didn't satisfy me because it's everything but a clean way. So I started some research for possibilities to pinvoke FFmpeg DLLs directly in .NET and after many disappointing projects which I coulnd't get to work I found FFmpeg.AutoGen which worked from the start with the example.

There are two parts in the solution: A .NET library to map ffpmeg DLLs and use them to extract frame images from a multimedia file and an application to easily create thumbnail sheets using the library.

The core of this project is based on the FFmpeg.AutoGen wrapper by Ruslan-B on GitHub (https://github.com/Ruslan-B/FFmpeg.AutoGen). I had to make a few changes mainly to support the meta data extration otherwise I would have imported the project directly.

To find out about the classes and functions needed to find information about a multimedia file, seek in a video stream and extract frame images I used the FFmpeg Documentation: https://www.ffmpeg.org/doxygen/trunk/

Using the application

The application can be used to create thumbnail sheets from multimedia files. There options to define how the sheet is designed. The following prameters can be set:

• The number of columns and rows of video frames
• The width of a single frame in pixel
• The margin around the whole sheet in pixel
• The padding between the frames (two times between header and frames) in pixel
• The background color
• The header font style and color
• The index (time position) font style, text and shadow color
• The border color of the frames and if it is to be drawn

There are also some options to configure how the application should akt:

• Close the applicatoin automatically when a passed job via commandline is finished
• Choose the filename of the image automatically (same as the movie file but image extention)
• The image format Bitmap, GIF, JPEG, PNG or TIFF (and quality if JPEG is selected)
• Use exact time positions or key-frames only

There are two ways to set those options: Using commandline parameters or an configuration file.

The configuration file is a standard MS INI file. You can edit or create the standard configuration file (same filename as the application but with .ini extention instead of .exe) by using the Settings dialog of the application or with a text editor. The section in which the options must be in is "SheetOptions". The following options are available:

These are the commandline parameters which override the settings from the configuration file:

Thumbnail sheets can be created by starting the application and then using the Open button to select one or more multimedia files or by drag'n'droping multimedia files to the application form; also passing a multimedia file via commandline parameter is possible:

ThumbSheetCreator.exe /I=0 /C=6 /R=2 /X=1 /F=1 /A=1 C:\MyMovie.mp4

Using the code

The main class is FFmpegMediaInfo and that's where you should take a look if you're planning to work with this code. You should initialize it with the multimedia file you wand to use. It will then collect some basic information about the file. Then the information and extraction methods can be used. Don't forget to dispose the class after you finished using it - or simply use "using":

// Load the multimedia file
using (FFmpegMediaInfo info = new FFmpegMediaInfo(@"C:\Videos\Test.mp4"))
{
    // Get the duration
    TimeSpan d = info.Duration;
    string duration = String.Format("{0}:{1:00}:{2:00}", d.Hours, d.Minutes, d.Second);

    // Get the video resolution
    Size s = info.VideoResolution;
    string resolution = String.Format("{0}x{1}", s.Width, s.Height);
    
    // Get the first video stream information
    FFmpegStreamInfo vs = info.Streams
        .FirstOrDefault(v => v.StreamType == FFmpegStreamType.Video);
    
    // If the video stream  exists, extract two random frames
    List<Bitmap> imgs = new List<Bitmap>();
    if (vs != null)
    {
        // Prepare random timestamps
        Random rnd = new Random();
        long dTicks = info.Duration.Ticks;
        TimeSpan t1 = new TimeSpan(Convert.ToInt64(dTicks * rnd.NextDouble()));
        TimeSpan t2 = new TimeSpan(Convert.ToInt64(dTicks * rnd.NextDouble()));
        
        // Extract images
        imgs = info.GetFrames(
            info.Streams.IndexOf(vs), // stream index
            new List<TimeSpan>() { t1, t2 }, // time positions of the frames
            true, // force exact time positions; not previous keyframes only
            (index, count) =>
            {
                // Get the progress percentage
                double percent = Convert.ToDouble(index) / Convert.ToDouble(count) * 100.0;
                
                return false; // Not canceling the extraction
            }
        );
    }

    // Extract a standard 6x5 frame thumbnail sheet from the default video stream
    Bitmap thumb = info.GetThumbnailSheet(
        -1, // Default video stream, will throw Exception if there is none
        new VideoThumbSheetOptions(6, 5), // preset sheet options with 6 columns and 5 rows
        (index, count) =>
        {
            // Get the progress percentage
            double percent = Convert.ToDouble(index) / Convert.ToDouble(count) * 100.0;
            
            return false; // Not canceling the extraction
        }
    );
}

FFmpegMediaInfo - the basic approach

If you need more functionality, feel free to edit the class FFmpegMediaInfo - it's not meant to be complete at all! Information on how to use the FFmpeg classes and functions can be found in the FFmpeg Documentation. The invoking is instance safe - means you can use more than one instance of the application at the same time; I'm not sure about threads within the same instance though as I haven't tried that yet. The code automatically uses the 32 or 64 bit version of FFmpeg depending on the platform type selected - even if "any platform" is selected, the 64 bit version will be used for suitable systems. Thus both versions should always be provided!

For a start here is the basic approach of the code: When a file is loaded (the function called is OpenFileOrUrl) the needed parts of FFmpeg are initialized. After that the file is loaded with FFmpeg and information about it are stored into an instanc of AVFormatContext. This class already contains most of the information extracted by FFmpegMediaInfo. The field nb_streams of AVFormatContext contains information about the different streams as type AVStream which are needed later on when extracting images. The metadata of as well AVFormatContext as AVStream are stored as type AVDictionary where some functions are needed to convert it to type Dictionary<string, string>. The Information as well as the AVFormatContext and AVStream instances are stored within the FFmpegMediaInfo instance.

The class is specialized on seeking to time positions. To avoid going through every single frame up to the passed time position, the keyframe prior to the time position is selected while seeking. To seek in a stream the function av_seek_frame() is used with setting skip_to_keyframe of the AVStream to 1 and the flag AVSEEK_FLAG_BACKWARD. Then (if ForceExactTimePosition is enabled) the next frames are decoded until the actual time position in the stream is within the time base (time per frame) or gets further from the seeked time position again. While seeking the field seek2any of the AVFormatContext must be set to 0 - otherwise the seeking will most likely end up in between two keyframes and up to the next keyframe the decoding will result in corrupt images because the frames in between are dependend on the prior keyframe. To extract frame images first the next package of teh selected video stream must be loaded using the function av_read_frame(), then the package must be decoded using the function avcodec_decode_video2(), the image parameters must be found using the function sws_scale() and after that everthing can be passed to the .NET Bitmap class to load the image. To determin the time position of the decoded frame the function av_frame_get_best_effort_timestamp() can be used.

Points of Interest

The most annoying thing about working with FFmpeg and pinvoking is that it's hard to find errors if there is one. I can only encourage you to try around and debug alot - Visual Studio is taking care of freeing the allocated memory if you break without disposing.

There is one major bug with using the keyframes mode that I haven't solved or even found yet: The first few pictures on a thumbnail sheet created the faster keyframe method way always seem to be the same and the timestamps are missing the targeted ones by quite a lot.

Future plans

To be honest, I don't have much time to work on this project, so don't expect updates soon! This is what I have in mind so far though:

• Fix the same image bug when using keyframes mode
• Implement an enumerator like seekable frame provider

Legal

The GNU Lesser General Public License v3 (LGPL3) for this article does apply to FFmpeg.AutoGen, FFmpeg and the code I've written. For FFmpeg the license is set to LGPL2.1 or later.