Gallery Server Pro - An ASP.NET Gallery for Sharing Photos, Video, Audio and Other Media

• Download source (external link)

Table of Contents

Introduction

Background

Running Gallery Server Pro

Solution Architecture

User Management and Security

Entity Framework Code First Migrations

A Template-Based User Interface

Rendering HTML for Video, Images, Audio and More

Transcoding video and audio with FFmpeg

Media Object, Albums and the Composite Pattern

Using the Strategy Pattern for Persisting to the Data Store

Summary

Introduction

Gallery Server Pro is a powerful and easy-to-use Digital Asset Management (DAM) application and web gallery for sharing and managing photos, video, audio, and other files. It is open source software released under GPL v3 and written with ASP.NET and C#. The entire application is contained in a single ASCX user control for easy insertion into your own web site.

• Organize your media files into albums you can easily add, edit, delete, rotate, sort, copy and move
• UI based on jsRender templates you can easily modify
• Uses HTML5 and CSS3 for plug-free video and audio, falling back to Flash and Silverlight when necessary
• Add files using one-click synchronize and ZIP file upload functions. Thumbnail and compressed versions are automatically created
• Supports external media objects such as YouTube videos
• Scalable to hundreds of thousands of objects
• Powerful user security with flexible, per-album granularity
• Metadata extraction
• Image watermarking
• SQL CE or SQL Server database
• 100% managed code written in C# and ASP.NET 4.5
• Source code is released under the open source GNU General Public License v3

You can play with an online demo of Gallery Server Pro to get a sense of its capabilities. A pre-compiled version is available, including additional documentation and a support forum.

Background

This project started in 2002 from my desire to share my photos over the web. I wanted my photos to remain on my own server, not somebody else's like Flicker or Facebook. Since there weren't any free solutions to choose from at the time, I wrote my own.

Version 1 was released to the world in January 2006. Since that time, there have been hundreds of thousands of downloads and a steady stream of releases. At the time of this writing, the latest version is 3.0.3. Check galleryserverpro.com for the latest version.

In this article, I present the overall architecture and technical features of Gallery Server Pro. The topics presented here can help if you want to learn more about:

• Implementing a web gallery to share photos, video, audio, and other documents
• Using Entity Framework Code First Migrations
• Rendering HTML with jsRender
• Creating an application that lets users modify jsRender templates from the site admin area
• Using FFmpeg to transcode video and audio from ASP.NET
• Using the composite design pattern to manage infinitely hierarchical relationships.
• When and how to use the strategy design pattern

Running Gallery Server Pro

Gallery Server Pro is a fully functional and stable web application ready for production use. To use the compiled version, visit the download page and follow the installation instructions in the Admin Guide.

Since you're reading this article, you may prefer to start with the source code. You'll need to use Visual Studio 2012 or higher. Here's how:

1. Download the source code. Unblock the ZIP file before extracting the contents. (Right-click the file in Windows Explorer, choose Properties, and then click 'Unblock' on the General tab. If you don't see this button, then ignore this message.)
2. Extract the directories and files to the desired location.
3. Start Visual Studio and open the solution file.
4. The code is configured to use a SQL CE database by default. If you want to use SQL Server, open web.config, uncomment the SQL Server connection string and edit it to provide credentials. Then comment out or delete the SQL CE connection string.
5. That's it! Press F5 to run the website. The gallery should automatically appear in the browser. Click the link to create an admin account. At this point you can use the gallery just as you would in a normal installation.

Gallery Server Pro stores media objects in a hierarchal set of albums. A media object is any type of file or a chunk of HTML code like the embed code you find on many websites. Media files and albums are stored in a directory named gs\mediaobjects within the web application. (This can be changed to any network-accessible location.) An album is really just a directory, so an album named Vacation Photos is stored as a similarly named directory.

There are two main techniques for adding media objects:

1. Upload a ZIP file containing the media files. If the ZIP file contains directories, they are converted to albums.
2. Copy your media files to the media objects directory, and then start a synchronization in Gallery Server Pro.

When adding a media object, the following steps occur:

1. The file is saved to the media objects directory. (If adding a media object via the synchronization technique, then this step is already done.)
2. Metadata about the file (eg. camera model, shutter speed, video length, etc) is extracted.
3. A thumbnail image is created and saved to the hard drive.
4. A compressed, bandwidth-friendly version is created for images, videos, and audio files.
5. A record is added to the data store to represent this media object.

Media objects are streamed to the browser through an HTTP handler. Below you can see a photo and a video being displayed. If watermarking is enabled, the watermark is applied to the in-memory version of the image just before it is sent.

The right pane shows metadata about the media object. The layout and editability of these items is highly configurable.

By default, everyone can browse the media objects. However, you must log on to perform any action that modifies an album or media object. Authorization to modify data is configured by type of permission and the albums to which it applies. For example, you can set up user Soren to have edit permission to the album Soren's photos. Another user Margaret is given edit permission to the album Margaret's photos. Each user can administer his or her own album but cannot edit albums outside his or her domain.

To learn more about how to use Gallery Server Pro from an end-user perspective, read the Administrator's Guide. Otherwise, read on to learn about the architecture and programming techniques.

Solution Architecture

The source code contains six projects. They are:

Project name	Description
Website	UI layer - ASP.NET 4.5 web application
TIS.GSP.Business	Business layer logic
TIS.GSP.Business.Resources	Contains resource data to support the business layer
TIS.GSP.Business.Interfaces	Defines all interfaces used in the solution
TIS.GSP.Events	Provides event handling support
TIS.GSP.Data.Data	Data layer logic

User Management and Security

User accounts are managed through the ASP.NET Membership and Roles APIs. By default, Gallery Server Pro is configured to use a locally stored SQL CE database named GalleryDb.sdf in the App_Data directory. It interacts with this database by using System.Web.Providers.DefaultMembershipProvider for users and System.Web.Providers.DefaultRoleProvider for roles.

Because of the flexibility offered by the provider model, you can use any data store that has a membership provider. For example, you can use SqlMembershipProvider to use SQL Server or ActiveDirectoryMembershipProvider to plug Gallery Server Pro into your existing base of Active Directory users. The Administrator's Guide contains a section for Membership Configuration that provides more information.

Entity Framework Code First Migrations

Overview

Versions of Gallery Server Pro prior to 3.0 used the data provider model for interacting with the database. This required creating classes that inherited from System.Configuration.Provider.ProviderBase to provide specific implementations for each supported data technology. This approach has some nice benefits, such as being able to target multiple databases and allowing each data implementation to take advantage of the strengths of the platform. For example, the SQL Server data provider used stored procedures for all data access.

However, this approach has a serious drawback—all data access code must be written once for each data provider. Since Gallery Server Pro shipped with two providers—SQL CE and SQL Server—that meant duplicating every piece of data access code. Of course, that doubled the testing as well. To further complicate things, the upgrade script that updated the data schema for each version had to be duplicated. This duplication of code violates the DRY principle, slows down development and is, frankly, not a lot of fun to develop and maintain.

So we switched to Entity Framework Code First Migrations for version 3.0. The primary benefit is that we write one chunk of LINQ code to interact with the database. The EF framework takes care of generating the correct SQL syntax for SQL CE or SQL Server. For example, to retrieve an album from the database we only need to write this:

using (var repo = new AlbumRepository())
{
  var album = repo.Where(a => a.AlbumId == albumId);
}

EF handles the rest for us, meaning we don't have to mess with ADO.NET commands, connections, or database-specific intricacies. Even better, migrations support means we can add a column to a table with database-agnostic code like this:

AddColumn("gsp.Metadata", "RawValue", c => c.String(maxLength: 1000)); 

When the gallery web application starts, EF initialization code checks the current version of the database, automatically upgrading it if necessary.

Database requirements

We had these requirements EF Code First Migrations had to handle:

• Automatically create the database, objects and initial data the first time the gallery starts or any time it is missing.
• If the database exists but doesn't yet have any gallery objects, create the objects and seed it with initial data.
• If an earlier version of the gallery data schema is present, automatically upgrade it to the current version. It must support upgrades from any previous version of Gallery Server Pro going back to 3.0.0.

These requirements must be simultaneously met without requiring changes to web.config or any other user input. As it turns out, it was a real challenge but we got there in the end.

Create and update the database

To have EF automatically manage the database, including creating, seeding, and updating as required, we run the following code during startup:

private static void InitializeDataStore()
{
  System.Data.Entity.Database.SetInitializer(new System.Data.Entity.MigrateDatabaseToLatestVersion<GalleryDb, GalleryDbMigrationConfiguration>());

  var configuration = new GalleryDbMigrationConfiguration();
  var migrator = new System.Data.Entity.Migrations.DbMigrator(configuration);
  if (migrator.GetPendingMigrations().Any())
  {
    migrator.Update();
  }
}

This code registers the MigrateDatabaseToLatestVersion database initializer and passes in a reference to the GalleryDbMigrationConfiguration class. Then it checks to see if there are pending migrations and, if so, executes them. The GalleryDbMigrationConfiguration class looks like this:

public sealed class GalleryDbMigrationConfiguration : DbMigrationsConfiguration<GalleryDb>
{
  protected override void Seed(GalleryDb ctx)
  {
    MigrateController.ApplyDbUpdates();
  }
}

The Seed method is called once all migrations are complete, so we can be guaranteed the database schema matches the Code First data model (assuming we've written the migration Up() methods correctly). This method is a good place to insert seed data or modify data that is required to support the new version. For example, the AppSetting table contains a record that stores the current data schema version (eg. "3.0.3"). When the gallery is updated to a newer version, this record must be updated as well.

At first one might think a good place for updating a record is in the migration's Up() method, and in fact early versions of Gallery Server Pro did this. However, this approach can fail when the schema is updated. For example, let's say we add a new column to the AppSetting table. The Up() method for the migration will include a call to AddColumn to add the new column, but the change won't actually propagate to the table until some point after exiting the Up() method. That means any attempt to query the AppSetting table from the Up() method will fail because the entity model (eg. the Code First AppSetting class) includes the new column but the database table doesn't.

This is why the Seed method is a good place to insert, update, or delete database records. We wrote a MigrateController class to handle these data updates:

public static void ApplyDbUpdates()
{
  using (var ctx = new GalleryDb())
  {
    if (!ctx.AppSettings.Any())
    {
      SeedController.InsertSeedData(ctx);
    }

    var curSchema = GetCurrentSchema(ctx);

    while (curSchema < GalleryDb.DataSchemaVersion)
    {
      var oldSchema = curSchema;

      switch (curSchema)
      {
        case GalleryDataSchemaVersion.V3_0_0: UpgradeTo301(); break;
        case GalleryDataSchemaVersion.V3_0_1: UpgradeTo302(); break;
        case GalleryDataSchemaVersion.V3_0_2: UpgradeTo303(); break;
        case GalleryDataSchemaVersion.V3_0_3: UpgradeTo310(); break;
      }

      curSchema = GetCurrentSchema(ctx);

      if (curSchema == oldSchema)
      {
        throw new Exception(String.Format("The migration function for schema {0} should have incremented the data schema version in the AppSetting table, but it did not.", curSchema));
      }
    }
  }
}

private static GalleryDataSchemaVersion GetCurrentSchema(GalleryDb ctx)
{
  return GalleryDataSchemaVersionEnumHelper.ConvertGalleryDataSchemaVersionToEnum(ctx.AppSettings.First(a => a.SettingName == "DataSchemaVersion").SettingValue);
}

The first thing the method does is insert seed data if no records are found in the AppSetting table. This will be the case when installing the gallery for the first time. The InsertSeedData() method populates several tables with lookup data and default settings. Refer to the source code if you're curious.

Next, the function gets the current data schema version from the AppSetting table in the database and uses it to call the corresponding update method. Refer to the source code to see these functions, but suffice it to say they update various tables like MediaTemplate and UiTemplate to implement bug fixes and update behavior. They also increment the data schema version in the AppData table.

We won't get into detail about how the migrations are created because that is pretty standard Code First Migrations stuff that is easily found on the internet. In short, we executed commands in the Package Manager Console of Visual Studio to enable migrations (Enable-Migrations) and add them (eg. Add-Migration v3.1.0). It is not necessary to call Update-Database because the the MigrateDatabaseToLatestVersion database initializer handles that for us the first time we run the application.

Specifying the database

You may have noticed the above node doesn't explicitly refer to SQL Server or SQL CE. Instead, it is database-agnostic and works equally well against either one. Well then, where do we specify which database technology we want to use and other relevant information such as the authentication info? This is all taken care of with the connection string in the web.config file in the root of the web application:

<connectionStrings>
  <clear />
  <add name="GalleryDb" providerName="System.Data.SqlClient" connectionString="server=(local);uid=;pwd=;Trusted_Connection=yes;database=GalleryServerPro;Application Name=Gallery Server Pro;MultipleActiveResultSets=True" />
</connectionStrings>

The DbContext in our application is named GalleryDb, so EF looks for a connection string with that name. In the example above, the connection string specifies the provider System.Data.SqlClient (SQL Server) and tells EF where the server is, how to authenticate, and the name of the database. When the application starts, EF looks for the database, creating it if necessary. Then it creates the database objects and seeds it with data. By the time the first page loads, you have a fully configured database and your gallery is ready to go.

If one wanted to use SQL CE instead, the only required change is an update to the connection string:

<connectionStrings>
  <clear />
  <add name="GalleryDb" providerName="System.Data.SqlServerCe.4.0" connectionString="data source=|DataDirectory|\GalleryData.sdf" />
</connectionStrings>

The next time the application starts, EF detects the updated connection string, creates the SQL CE database, seeds it with data, and begins using it.

This behavior is really nice during development, because it means one can delete the database at any time and have it auto-created during the next page load. Since this is the same process that occurs for end users when they install the gallery, it is a convenient way to test the installation process to verify that changes in default settings or other setup behavior is occurring as expected.

NOTE: If you place an empty text file named install.txt in the App_Data directory, the startup behavior changes slightly. A link will appear on the main page that lets you create an administrator account. Since you need an admin account to manage the gallery, this is typically something you'll want to do at the same time you have EF create the database.

A Template-Based User Interface

Overview

UI templates are a powerful feature that lets you modify the user interface using only HTML and JavaScript skills. For example, here are a few things you can do with UI templates:

• Add your company logo to the header
• Add an image preview when hovering over a thumbnail
• Add links in the album tree to common tags
• Show the five most recently added items at the top of an album

We'll look at how to do all these examples—and a few more—later in this section. But first let's get the basics down. There are five sections in the gallery that are built from UI templates:

These images show the five templates: header, left pane, right pane, album, and media object. The center pane shows either the album or media object template, depending on whether the user is looking at an album or an individual media object.

Each section shown above is rendered from a UI template. A UI template consists of a chunk of HTML containing jsRender syntax and some JavaScript that tells the browser what to do with the HTML. Typically the script invokes the jsRender template engine and appends the generated HTML to the page. A robust set of gallery data is available on the client that gives you access to important information such as the album and media object data, user permissions, and gallery settings. We'll get into the structure of this data later in this post.

You can view the UI template definitions on the UI Templates page in the site admin area. In these two images you can see the HTML and JavaScript values:

You can assign which albums this particular template applies to on the Target Albums tab. If multiple templates target the same album, the most specific one wins. For example, if the default template is assigned to 'All albums' and a second template is assigned to the Samples album, the second template will be used for the Samples album and any of its children because that template definition is 'closer' to the album.

The preview tab lets you see the result of any edits you make before saving the changes.

Anatomy of the left pane

Let's take a close look at how one of the templates works. We'll choose the left pane template first. The HTML is simple:

<div id='{{:Settings.ClientId}}_lptv'></<div>

It defines a single, empty div tag and gives it a unique ID. The text between the double brackets is jsRender syntax that refers to the ClientId property of the Settings object. This particular property provides a string that is unique to the current instance of the Gallery user control on the page. When it is rendered on the page you end up with HTML similar to this:

<div id='gsp_g_lptv'></<div>

Note: Defining a unique ID is not required for most galleries, but it is there for admins who want to include two instances of the gallery control on a page. For example, you might have a slideshow running in one part of a web page and a video playing in another.

Astute observers will notice that a single div tag doesn't look anything like a complex treeview. So how does that div tag eventually become the treeview? Let's look at the JavaScript that is part of the template:

// Render the left pane, but not for touchscreens UNLESS the left pane is the only visible pane
var isTouch = window.Gsp.isTouchScreen();
var renderLeftPane = !isTouch  || (isTouch && ($('.gsp_tb_s_CenterPane:visible, .gsp_tb_s_RightPane:visible').length == 0));

if (renderLeftPane ) {
 $('#{{:Settings.LeftPaneClientId}}').html( $.render [ '{{:Settings.LeftPaneTmplName}}' ]( window.{{:Settings.ClientId}}.gspData ));

 var options = {
  albumIdsToSelect: [{{:Album.Id}}],
  navigateUrl: '{{:App.CurrentPageUrl}}'
 };

 // Call the gspTreeView plug-in, which adds an album treeview
 $('#{{:Settings.ClientId}}_lptv').gspTreeView(window.{{:Settings.ClientId}}.gspAlbumTreeData, options);
}

Technically this text is not pure JavaScript. See the jsRender syntax in there? That's right, it is ALSO a jsRender template that will be run through the jsRender engine to produce pure JavaScript just before execution. This is an extraordinarily powerful feature and can be harnessed to produce a wide variety of UI possibilities. Imagine writing script that loops through the images in an album to calculate some value or invoke a callback to the server to request data about a specific album.

The first thing the script does is decide whether the left pane template should even be appended to the page. We decide not to show it on touchscreens for two reasons: (1) touchscreens typically have small screens and cannot afford the real estate required by the left pane (2) the splitter control that separates the left, center, and right panes does not work well on touchscreens.

The script refers to the function window.Gsp.isTouchScreen(). You'll find this function in the minified file gs\script\gallery.min.js. If you are editing the gallery you will probably prefer to have un-minified script files loaded into the browser. You can easily accomplish this by switching the debug setting to 'true' in web.config.

If the script decides the left pane is to be rendered, it executes this line:

$('#{{:Settings.LeftPaneClientId}}').html( $.render [ '{{:Settings.LeftPaneTmplName}}' ]( window.{{:Settings.ClientId}}.gspData ));

This will look familiar to anyone with jsRender experience or other types of template engines. In plain English, it says to take the template having the name LeftPaneTmplName and the data in the variable gspData, run in through the jsRender engine, and assign the resulting HTML to the HTML element named LeftPaneClientId. In other words, it takes the string

    <div
id='{{:Settings.ClientId}}_lptv'></div>

, converts it to <div id='gsp_g_lptv'></div>, and adds it to the page's HTML DOM.

So far all the script has done is add a div tag to the page, but we still need to convert it into the album tree. That's what the last section does:

var options = {
 albumIdsToSelect: [{{:Album.Id}}],
 navigateUrl: '{{:App.CurrentPageUrl}}'
};

// Call the gspTreeView plug-in, which adds an album treeview
$('#{{:Settings.ClientId}}_lptv').gspTreeView(window.{{:Settings.ClientId}}.gspAlbumTreeData, options);

The JavaScript file I mentioned above contains several jQuery plug-ins to help reduce the complexity of the templates and to maximize the amount of script that is cached by the browser (inline script is never cached). Here we use jQuery to grab a reference to our generated div tag and we invoke the gspTreeView plug-in on it, passing along the album treeview data and a few options. The tree data is a JSON object containing the album structure and is included in every page request. In turn, the gspTreeView plug-in is a wrapper around the third party jQuery tree control jsTree, the code for which is in the lib.min.js file (or lib.js if you are running with debug=true). There are several third party script libraries in that file.

The treeview plug-in builds an HTML tree from the data and appends it to the div tag, resulting in the tree view you see in the left pane. You can play with the HTML and JavaScript and then use the preview tab to see how those edits affect the output.

Client data model

Each page contains a rich set of data that is included with the browser request. A global JavaScript variable named gspData exists that is scoped to the current gallery user control instance. In a default installation, you can find it at window.gsp_g.gspData, as seen in this image from Chrome:

Let's take a brief look at each top-level property:

ActiveGalleryItems – An array of GalleryItem instances that are currently selected. A gallery item can represent an album or media object (photo, video, audio, document, YouTube snippet, etc.)

ActiveMetaItems – An array of MetaItem instances describing the currently selected item(s).

Album – Information about the current album. It has two important properties worth explaining: GalleryItems and MediaItems. Both represent albums and media objects, but a GalleryItem instance contains only basic information about each item while a MediaItem instance contains all the metadata and other details about an item. Because a GalleryItem instance is lightweight, it is well suited for album thumbnail views where you only need basic information. Therefore, to optimize the data passed to the browser, the MediaItems property is null when viewing an album and the GalleryItems property is null when viewing a single media object.

App – Information about application-wide settings, such as the skin path and current URL.

Resource – Contains language resources.

Settings – Contains gallery-specific settings.

User – Information about the current user.

Here is a complete list of client objects and their properties (click images for a larger version):

More documentation for the client API can be found in the Administrator's Guide (look in the UI Templates section) and the GalleryServerPro.Web.Entity API documentation.

Rendering HTML for Video, Images, Audio and More

Overview

Rendering JPG images in a browser is straightforward because all browsers use the same syntax based on the <img> HTML tag. But video, audio, and other file types have wildly different levels of support in various browsers and within different versions of the same browser. For example, some browsers can natively play H.264 MP4 video while others require a plug-in such as Flash or Silverlight.

Gallery Server Pro solves this problem with a set of media templates that can deliver the correct HTML and JavaScript for individual browsers or even versions of browsers. A default set of templates are included that will work well for most users, but some organizations may require different behavior. Or, when a new browser version is released, its support for a particular media type may change, requiring a change in the HTML syntax that is used.

The media templates page lets one manage the rendering behavior of different media types.

The above screen shot shows the media template used for the 'default' browser when rendering a media file having MIME type video/mp4. Notice it specifies the HTML5 <video> tag. When MP4 videos are viewed in most browsers, the HTML is based on this template, as seen here in Safari:

The Browser ID dropdown allows one to view media templates for other browsers. For example, the MP4 templates includes two additional templates to support IE 1-8 and Opera, which do not support the HTML5 video tag:

Notice the HTML template for IE 1-8 and Opera are identical and consists of a hyperlink tag. This defines the HTML for using FlowPlayer, which uses the Flash plug-in to render the video. FlowPlayer requires some JavaScript to initialize it, which you can see when you click the JavaScript tab:

When an MP4 video is viewed in IE 1-8 or Opera, it gets HTML and JavaScript based on the corresponding template, as seen in this screen shot from Opera:

Through the use of custom media templates for individual browsers, Gallery Server Pro can provide targeted behavior without requiring complicated HTML containing fallback code and other tricks. The Administrator's Guide has more information about editing and creating media templates.

Transcoding video and audio with FFmpeg

Overview

FFmpeg is an open source utility that can transcode video and audio and extract thumbnail images from videos. When ffmpeg.exe is present in the bin directory, Gallery Server Pro automatically uses it to create web-optimized versions of video and audio files. This can result in dramatically smaller files and an improved user experience.

FFmpeg can be downloaded from the official FFmpeg website. Since the site posts only the source code, you have to compile it yourself. For convenience, we've done that part and packaged it into a free download called the Gallery Server Pro Binary Pack. Although we won't talk about them here, the package includes two additional open source tools—ImageMagick and GhostScript—which are used to generate thumbnails from certain image and document types.

Using FFmpeg is easy. For example, one can create a Flash Video file from a Windows Media File video by executing the following at a command prompt:

ffmpeg.exe -i "C:\myvideo.wmv" "C:\myvideo.flv"

FFmpeg supports a large number of options that control how the output video or audio file is created. In Gallery Server Pro, we want to create H.264 MP4 web-optimized videos that will play in as many devices as possible, including mobile devices. We came up with the following:

 ffmpeg.exe -y -i "{SourceFilePath}" -vf "scale=min(iw*min(640/iw\,480/ih)\,iw):min(ih*min(640/iw\,480/ih)\,ih)" -b:v 384k -vcodec libx264 -flags +loop+mv4 -cmp 256 -partitions +parti4x4+parti8x8+partp4x4+partp8x8 -subq 6 -trellis 0 -refs 5 -bf 0 -coder 0 -me_range 16 -g 250 -keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 -qmin 10 -qmax 51 -qdiff 4 -ac 1 -ar 16000 -r 13 -ab 32000 -movflags +faststart "{DestinationFilePath}"

That's a real handful to understand and we won't disect the entire thing, but let's look at a few key things. -vf specifies a filter that resizes the video to approximately 640px by 480px while preserving the aspect ratio. -vcodec specifies the libx264 codec. -movflags uses the faststart option that results in a video that can be played after a small amount of buffering. Without this flag, the entire video must be downloaded before it begins playback.

Learn more about these options by reading the FFmpeg documentation. The options used in Gallery Server Pro can be viewed and changed on the Video & Audio page in the site admin area.

Calling FFmpeg from ASP.NET

As we just saw, we can use FFmpeg to transcode a video or audio file at a command prompt. To invoke it from ASP.NET, we use System.Diagnostics.Process:

private void Execute()
{
  bool processCompletedSuccessfully = false;
  
  InitializeOutput();
  
  var info = new ProcessStartInfo(AppSetting.Instance.FFmpegPath, MediaSettings.FFmpegArgs);
  info.UseShellExecute = false;
  info.CreateNoWindow = true;
  info.RedirectStandardError = true;
  info.RedirectStandardOutput = true;
  
  using (Process p = new Process())
  {
    try
    {
      p.StartInfo = info;
      // For some reason, FFmpeg sends data to the ErrorDataReceived event rather than OutputDataReceived.
      p.ErrorDataReceived += ErrorDataReceived;
      //p.OutputDataReceived += new DataReceivedEventHandler(OutputDataReceived);
      p.Start();
  
      p.BeginErrorReadLine();
  
      processCompletedSuccessfully = p.WaitForExit(MediaSettings.TimeoutMs);
  
      if (!processCompletedSuccessfully)
        p.Kill();
  
      p.WaitForExit();
  
      if (!processCompletedSuccessfully || MediaSettings.CancellationToken.IsCancellationRequested)
      {
        File.Delete(MediaSettings.FilePathDestination);
      }
    }
    catch (Exception ex)
    {
      if (!ex.Data.Contains("args"))
      {
        ex.Data.Add("args", MediaSettings.FFmpegArgs);
      }
  
      Events.EventController.RecordError(ex, AppSetting.Instance, MediaSettings.GalleryId, Factory.LoadGallerySettings());
    }
  }
}

/// <summary>
/// Seed the output string builder with any data from a previous conversion of this
/// media object and the basic settings of the conversion.
/// </summary>
private void InitializeOutput()
{
  var item = MediaConversionQueue.Instance.GetCurrentMediaQueueItem();
  if ((item != null) && (item.MediaQueueId == MediaSettings.MediaQueueId))
  {
    // Seed the log with the existing data; this will prevent us from losing the data
    // when we save the output to the media queue instance.
    _output.Append(item.StatusDetail);
  }

  IMediaEncoderSettings mes = MediaSettings.EncoderSetting;
  if (mes != null)
  {
    _output.AppendLine(String.Format("{0} => {1}; {2}", mes.SourceFileExtension, mes.DestinationFileExtension, mes.EncoderArguments));
  }

  _output.AppendLine("Argument String:");
  _output.AppendLine(MediaSettings.FFmpegArgs);
}

/// <summary>
/// Handle the data received event. Collect the command line output and cancel if requested.
/// </summary>
/// <param name="sender">The sender.</param>
/// <param name="e">The <see cref="System.Diagnostics.DataReceivedEventArgs"/> instance 
/// containing the event data.</param>
private void ErrorDataReceived(object sender, DataReceivedEventArgs e)
{
  _output.AppendLine(e.Data);

  var item = MediaConversionQueue.Instance.GetCurrentMediaQueueItem();
  if ((item != null) && (item.MediaQueueId == MediaSettings.MediaQueueId))
  {
    item.StatusDetail = Output;
    // Don't save to database, as the overhead the server/DB chatter it would create is not worth it.
  }

  CancelIfRequested(sender as Process);
}

/// <summary>
/// Kill the FFmpeg process if requested. This will happen when the user deletes a media
/// object that is being processed or deletes the media queue item in the site admin area.
/// </summary>
/// <param name="process">The process running FFmpeg.</param>
private void CancelIfRequested(Process process)
{
  CancellationToken ct = MediaSettings.CancellationToken;
  if (ct.IsCancellationRequested)
  {
    if (process != null)
    {
      try
      {
        process.Kill();
        process.WaitForExit();
      }
      catch (Win32Exception) { }
      catch (SystemException) { }
    }
  }
}

The full path to the ffmpeg.exe executable and the command line arguments are passed into an instance of ProcessStartInfo, which is then assigned to the StartInfo property of a Process instance. As FFmpeg processes the file, it periodically returns data, which is collected in a StringBuilder in the ErrorDataReceived event and ultimately logged to the event log.

A timeout value is passed to the WaitForExit method to automatically shut down the process if it doesn't complete in a reasonable time.

Killing the process

There are a couple situations where we want to kill the process before it completes on its own:

1. The user deletes the media object being processed.
2. The user deletes the media queue item on the Video & Audio page in the site admin area.

Each time FFmpeg calls the ErrorDataReceived event to update it with a progress report, we check a cancellation token to see if the flag has been set to request a cancellation. If it has, we call the Kill method as shown in the code snippet above.

But how is the cancellation token assigned and updated? The processing of media items with FFmpeg is managed by a singleton class named MediaConversionQueue. That class has a property called CancelTokenSource:

protected CancellationTokenSource CancelTokenSource { get; set; }

Each time we ask FFmpeg to do a conversion, we assign this property and pass its Token property to the FFmpeg class:

CancelTokenSource = new CancellationTokenSource();

var mediaSettings = new MediaConversionSettings
  {
    // ...Other stuff ommitted for clarity...
    CancellationToken = CancelTokenSource.Token
  };

mediaSettings.FFmpegOutput = FFmpeg.CreateMedia(mediaSettings);

This token is what we are checking in the ErrorDataReceived event. When a media object is deleted or the queue item is removed, the Cancel method is invoked on the token source (in MediaConversionQueue.RemoveMediaQueueItem):

CancelTokenSource.Cancel();

Calling the Cancel method causes subsequent calls to the IsCancellationRequested property to return true, triggering the code in the ErrorDataReceived event to kill the process.

Media Object, Albums and the Composite Pattern

Each media object (photo, video, etc.) is stored in an album. Albums can be nested within other albums, with no restriction on the number of levels. This is similar to how files and directories are stored on a hard drive.

It turns out that albums and media objects have a lot in common. They both have properties such as Id, Title, DateAdded, and FullPhysicalPath; and they both have methods such as Save and Delete. This is the ideal situation in which to use the composite design pattern, where common functionality is defined in a base object. I start by defining two interfaces — IGalleryObject and IAlbum:

The IAlbum interface inherits from IGalleryObject and then adds a few methods and properties that are specific to albums. Then I create the abstract base class GalleryObject. It implements the IGalleryObject interface and provides default behavior that is common to albums and media objects. For example, here is the DateAdded property:

public DateTime DateAdded
{
  get
  {
    VerifyObjectIsInflated(this._dateAdded);
    return this._dateAdded;
  }
  set
  {
    this._hasChanges = (this._dateAdded == value ? this._hasChanges : true);
    this._dateAdded = value;
  }
}

Now that the common functionality is defined in the abstract base class, I can create concrete classes to represent albums, images, video, audio, and other types of media objects:

With this approach, there is very little duplicate code, the structure is maintainable, and it is easy to work with. For example, when Gallery Server Pro wants to display the title and thumbnail image for all the objects in an album, there might be any combination of child albums, images, video, audio, and other documents. But I don't need to worry about all the different classes or about casting problems. All I need is the following code:

// Assume we are loading an album with ID=42

IAlbum album = Factory.LoadAlbumInstance(42, true);
foreach (IGalleryObject galleryObject in album.GetChildGalleryObjects())
{
  string title = galleryObject.Title;
  string thumbnailPath = galleryObject.Thumbnail.FileNamePhysicalPath;
}

Beautiful, isn't it? But what happens when the functionality is slightly different between two types of objects? For example, Gallery Server Pro needs to verify that each media object has a thumbnail image, but albums do not have a thumbnail image, at least not in the strict sense of the word. (The thumbnail image you see for albums is actually the thumbnail of one of its media objects.) We want to define this validation in the base class so that it is inherited by all concrete classes, but one of the concrete classes—Album—requires different validation behavior.

We solve this by first creating the validation function in the GalleryObject class:

protected virtual void CheckForThumbnailImage()
{
  if (!System.IO.File.Exists(this.Thumbnail.FileNamePhysicalPath))
  {
    this.RegenerateThumbnailOnSave = true;
  }
}

Since it is defined as protected virtual, we can override it in a derived class. In fact, that is exactly what the Album class does:

protected override void CheckForThumbnailImage()
{
  // Do nothing
}

The end result is we have an implementation in the base class that provides functionality for most cases and code that is unique to albums is contained in the Album class. There isn't any duplicate code and the logic is nicely encapsulated. It is a thing of beauty to behold.

Using the Strategy Pattern for Persisting to the Data Store

We just saw how to override a function when we need to alter its behavior. We could have done something similar when it comes to saving the albums and media objects to the database. The Save method in the GalleryObject class could have been defined as virtual, and we could have overridden the method in each of the derived classes. But since the classes Image, Video, Audio, GenericMediaObject, and ExternalMediaObject all represent objects that get stored in the same table (MediaObject), that would have meant writing the same code in all five classes, with only the Album class being different.

One approach that eliminates the problem of duplicate code is to provide a default implementation in the Save method in the GalleryObject class. In that method, I save to the media object table, and then depend on the Album class to override the behavior, much like we did with the CheckForThumbnailImage function. However, this is putting a substantial amount of behavior in a base class that doesn't really belong there. We should limit the base class to contain state and behavior that applies to ALL derived objects.

You might argue that I violated this rule when I provided a default implementation of the CheckForThumbnailImage method that I overrode in the Album class. You are absolutely right. But I justify it by suggesting that implementing the validation in every derived class creates undesirable duplicate code, and refactoring it to use the strategy pattern is overkill. These are not hard and fast rules. Architecting an application is as much art as it is science, and you must weigh the pros and cons of each approach.

Getting back to our challenge of persisting data to the data store, the approach we came up with was to use the strategy pattern to encapsulate behavior. First, I defined an interface ISaveBehavior :

public interface ISaveBehavior { void Save(); }

Then I wrote two classes that implemented the interface: AlbumSaveBehavior and MediaObjectSaveBehavior. The Save method takes care of persisting the object to the hard drive and data store. For example, here is the Save method in AlbumSaveBehavior:

public void Save()
{
  if (this._albumObject.IsVirtualAlbum)
    return; // Don't save virtual albums.

  // Must save to disk first, since the method queries properties that might 
  // be updated when it is saved to the data store.
  PersistToFileSystemStore(this._albumObject);

  // Save to the data store.
  using (var repo = new AlbumRepository())
  {
    repo.Save(this._albumObject);
  }

  if (this._albumObject.GalleryIdHasChanged)
  {
    // Album has been assigned to a new gallery, so we need to iterate through all
    // its children and update those gallery IDs as well.
    AssignNewGalleryId(this._albumObject.GetChildGalleryObjects(GalleryObjectType.Album), this._albumObject.GalleryId);
  }
}

We'll skip showing the implementation from the MediaObjectSaveBehavior class, but it contains logic that is specific to persisting media objects.

OK, we have two classes for saving data to the data store—one for albums and one for media objects. How do we invoke the appropriate Save method from the GalleryObject base class?

Recall that the GalleryObject class is abstract, so it can never be directly instantiated. Instead, we instantiate an instance of the Album, Image, Video, Audio, GenericMediaObject, or ExternalMediaObject class. The constructor for each of these classes assigns the appropriate save behavior. For example, in the constructor of the Album class, we have:

this.SaveBehavior = Factory.GetAlbumSaveBehavior(this);

The GetAlbumSaveBehavior method just returns an instance of the AlbumSaveBehavior class:

public static ISaveBehavior GetAlbumSaveBehavior(IAlbum albumObject)
{
  return new AlbumSaveBehavior(albumObject);
}

The SaveBehavior property of the GalleryObject class is of type ISaveBehavior. Since both classes implement this interface, we can assign instances of either class to the property.

The Save method in the GalleryObject class simply calls the Save method on the

SaveBehavior

property. It has no idea whether the property is an instance of AlbumSaveBehavior or MediaObjectSaveBehavior, and it doesn't care. All that matters is that each class knows how to save its designated object.

This is an example of using the strategy pattern. Specifically, the strategy pattern is defined as a family of algorithms that are encapsulated and interchangeable. In our case, we have two save behaviors that are self-contained and can both be assigned to the same property (interchangeable). It is a powerful pattern and has many uses.

Summary

This has been a brief introduction to the architecture and programming techniques used in Gallery Server Pro. Feel free to download the source code and use the bits to help in your own project. Cheers!

Article History

• 2013 Oct 18
  • Updated for version 3, including several new sections
  • Updated source code to 3.0.3
• 2011 July 1
  • Updated source code
• 2011 Apr 27
  • Updated source code to 2.4.7
  • Updated article to reflect changes since last update
• 2008 Nov 3
  • Updated source code to 2.1.3222
• 2008 Sep 9
  • Updated to include latest source files, references to new features, and minor content updates
• 2008 May 6
  • Updated to include latest source files and minor content updates
• 2007 Oct 28
  • Article release