Image in Azure: Upload, Deep Zoom Viewing, Mark Out and Annotation

• Download source - 117.69 KB

Introduction

One obvious application of cloud computing is providing storage, processing and fast access to large volume of data, particularly images. Dealing for some time with a non-cloud-based image access system, I'd like to implement a simple software allowing to upload images to Azure Blob Storage and then easily view them. In addition to fast access to the stored image, it is important to let user to mark particular areas out of the image, annotate them and store this information in Azure Blob Storage for future use. It is preferred to provide upload, viewing and mark out using only browser with minimum (ideally zero) installation on client machine. Bearing this goal in mind, I dug the Web for relevant information and samples. I found a lot of writings covering some aspects of the problem, mainly related to image viewing. I didn't come across a complete code sample addressing the above problem. Nor did I find a suitable implementation of image mark out.

This article presents code sample capable of:

• uploading image from local machine / device to Azure Blob Storage,
• instant download of the stored image with possibility to scale, size and move it with almost zero delay,
• marking particular areas of the image out and annotating them, and
• saving / loading mark out and annotation metadata to / from Azure Blob Storage along with the image.

My code is based on the previous work of other people (please see references at the end of the article), particularly on the CodeProject article by Joerg Lang [9].

Background

From a user perspective, the software provides the possibility to upload image file (currently jpg and png formats are supported by viewer) from any machine or device via appropriate Web page to Windows Azure Blob Storage, and then using another Web page, to view the image, mark specific areas (currently rectangular ones only) out on the image, write descriptions (annotation, remarks) about the marked areas and save the areas along with their descriptions.

It seems logical that file upload should be carried out from a simple HTML page readable by any browser. This allows the user to upload image from virtually any computer and mobile device. WCF RESTful service ImagingSvc acts as a counterpart on the server (Azure cloud) side. This service is responsible for providing HTML page for upload the image to Azure. For image viewing, MultiScaleImage control of Microsoft Silverlight is employed. So the viewing browser should support Silverlight. Currently most of the browsers do meet this requirement (except probably browser of mobile devices). MultiScaleImage control is based on the Deep Zoom (DZ) technology [1-6]. Wikipedia describes DZ as following [1]:

Deep Zoom is an implementation of the open source technology, provided by Microsoft, for use in for image viewing applications. It allows users to pan around and zoom in a large, high resolution image or a large collection of images. It reduces the time required for initial load by downloading only the region being viewed and/or only at the resolution it is displayed at. Subsequent regions are downloaded as the user pans to (or zooms into them); animations are used to hide any jerkiness in the transition.

DZ requires pyramid of tiles images constructed from original image [2, 3]. The image pyramid exceeds size of original image (according to some estimation in 1.3 time in average). But this technique permits fast and smooth image download for viewing. Microsoft provides a special Deep Zoom Composer [7] tool for the tile image pyramid generation. But usage of this tool does not help in our task since this first requires installation of the tool, and second, considerably increases volume of data to be uploaded. Clearly, we have to provide a more suitable to our purposes tool for tile pyramid generation.

Another problem to be solved is that MultiScaleImage control being an excellent tool for image viewing does not provide support to mark out and annotation tasks. This means that these important features should be implemented "manually". In context of the features implementation, it is important not only to draw frames for chosen areas, but also synchronize movements of these frames and underlying image.

Design

The CodeProject articles [8, 9] and blog entry [10] were chosen as departure points for the design. The former provide algorithm and code for image tile pyramid generation. The change required was to store the images as blobs in the Azure Blob Storage rather then in database. The latter contains useful tips to set permissions to Azure Blob Storage container. While developing, I also used Cloudberry Explorer tool to inspect and manage Azure Blob Storage.

A Web Role WCF application ImagingService was created with VS2010 wizard. Two WCF service components ImagingSvc and SlSvc, and a Silverlight components MsiHelperLib and SlImage were added to the application. The VS2010 solution structure is depicted in the figure below:

The RESTful WCF service ImagingSvc provides a means for uploading image from client machine / device, its processing to a DZ image tile pyramid and storage of the image pyramid to Azure Blob Storage. This service has webHttpBinding. SlSvc WCF service intends to internal-to-Azure-application communication to provide data for SlImage Silverlight-based component. The service has basicHttpBinding and a relative address to simplify access to it by SlImage component. Interfaces and configuration of both WCF services are shown below:

[ServiceContract]
interface IImagingSvc
{
    [OperationContract]
    [WebGet(UriTemplate = "/{data}", BodyStyle = WebMessageBodyStyle.Bare)]
    Message Init(string data);

    [OperationContract]
    [WebInvoke(Method = "POST", BodyStyle = WebMessageBodyStyle.Bare)]
    Message Upload(Stream stream);
}

[ServiceContract]
public interface ISlSvc
{
    [OperationContract]
    string BlobContainerUri();

    [OperationContract]
    string[] BlobsInContainer();

    [OperationContract]
    void SaveEditing(string blobId, string xmlEditing);

    [OperationContract]
    string LoadEditing(string blobId);
}

<configuration>
    <system.diagnostics>
    	<trace>
    	    <listeners>
                <add type="Microsoft.WindowsAzure.Diagnostics.
		DiagnosticMonitorTraceListener, Microsoft.WindowsAzure.
		Diagnostics, Version=1.0.0.0, Culture=neutral, 
		PublicKeyToken=31bf3856ad364e35"
                           name="AzureDiagnostics">
                    <filter type="" />
                </add>
            </listeners>
        </trace>
    </system.diagnostics>

    <system.web>
        <compilation debug="true" targetFramework="4.0" />
        <httpRuntime maxRequestLength="2147483647"/>
    </system.web>

    <system.serviceModel>
        <serviceHostingEnvironment multipleSiteBindingsEnabled="true" />
        <bindings>
            <webHttpBinding>
                <binding name="StreamWebHttpBinding"
                    maxBufferPoolSize="1000"
                    maxBufferSize="2147483647" maxReceivedMessageSize="2147483647"
                    closeTimeout="00:25:00" openTimeout="00:01:00"
                    receiveTimeout="01:00:00" sendTimeout="01:00:00"
                    transferMode="Streamed" bypassProxyOnLocal="false">
                        <readerQuotas maxDepth="333333" maxStringContentLength="333333"
                             maxArrayLength="333333"
                             maxBytesPerRead="333333" maxNameTableCharCount="333333" />
               </binding>
           </webHttpBinding>
       </bindings>

       <behaviors>
           <endpointBehaviors>
               <behavior name="RestBehavior">
                   <webHttp />
               </behavior>
           </endpointBehaviors>
           <serviceBehaviors>
               <behavior name="LargeUploadBehavior">
                   <serviceDebug includeExceptionDetailInFaults="true" />
               </behavior>
               <behavior name="">
                   <serviceMetadata httpGetEnabled="true" />
                   <serviceDebug includeExceptionDetailInFaults="false" />
               </behavior>
           </serviceBehaviors>
       </behaviors>

       <services>
           <service name="ImagingService.ImagingSvc"
           behaviorConfiguration="LargeUploadBehavior" >
               <endpoint address=""
               contract="ImagingService.IImagingSvc"
               binding="webHttpBinding"
               bindingConfiguration="StreamWebHttpBinding"
               behaviorConfiguration="RestBehavior" />
           </service>

           <service name="ImagingService.SlSvc" >
               <endpoint address=""
               contract="ImagingService.ISlSvc" binding="basicHttpBinding" />
           </service>
       </services>
    </system.serviceModel>

    <system.webServer>
        <modules runAllManagedModulesForAllRequests="true"/>
    </system.webServer>
</configuration>

MsiHelperLib.MsiHelper type works as a wrapper around MultiScaleImage managing this control and linking it to main SlImage component. Constructor of MsiHelper class takes three GUI controls defined in main GUI component (SlImage is our case), namely, MultiScaleImage, canvas and DataGrid as parameters and coordinates their behavior and events. MsiHelper is responsible for major activities with MultiScaleImage, like mouse button and wheel events for image zooming and movement, as well as handling of mark out, annotation and serialization of related data. Some of its methods are made virtual protected to facilitate addition of new features in derived classes during further development. Class MouseWheelHelper manages mouse wheel, while DescriptionChildWindow provides description (annotation) dialog form.

SlImage is the main Web GUI component for viewing previously uploaded and processed images. It acts as client for SlSvc WCF service. SlImage defines MultiScaleImage, canvas and DataGrid controls and uses MsiHelperLib component to manage them. Its MainPage type handles events caused by GUI controls and calls appropriate methods of MsiHelperLib.MsiHelper type. SlImage is publicly accessed via dedicated SlImage.html file. Being a Silverlight-based, object SlImage cannot communicate with Azure objects directly. For this communication (e.g. to get a list of stored image blobs) SlImage relies on SlSvc WCF service. VS2010 allows the developer to easily create Service Reference on SlSvc WCF service and generate appropriate class SlImage.SlSvc_ServiceReference.SlSvcClient. SlImage uses instance of this class (proxy) for communication with main application. Tag <configuration> <system.servicemodel> <bindings> <system.serviceModel><system.servicemodel> of SlImage configuration file ServiceReferences.ClientConfig is shown below:

<configuration><system.servicemodel><bindings><system.serviceModel>
    <bindings>
        <basicHttpBinding>
            <binding name="BasicHttpBinding_ISlSvc"
            maxBufferSize="2147483647" maxReceivedMessageSize="2147483647">
                <security mode="None" />
            </binding>
        </basicHttpBinding>
    </bindings>
    <client>
        <endpoint address="../SlSvc.svc"
        binding="basicHttpBinding" contract="SlSvc_ServiceReference.ISlSvc" />
    </client>
</system.serviceModel><basichttpbinding>
<binding name="BasicHttpBinding_ISlSvc" maxBufferSize="2147483647" 
	maxReceivedMessageSize="2147483647">
</binding></basichttpbinding></bindings>
</system.servicemodel></configuration>

Mark Out

As it was stated above, code for this article allows user to mark selected areas of interest out in the image. Instance of System.Windows.Controls.Canvas type is used to draw mark out frames (current version supports rectangular areas only, but MsiHelper type infrastructure and its virtual methods facilitate handling of areas with more complex boundaries in derived classes). Implementation of drawing mark out rectangle on the canvas is a relatively simple task. As it had been turned out, much more complex problem is to synchronize behavior of this mark out with image itself during various visual transformations. The following code shows geometric transform to ensure synchronic visual evolutions:

public CompositeTransform MsiTransform
{
    get
    {
    	CompositeTransform compositeTransform = new CompositeTransform();
        compositeTransform.TranslateX = msi.Margin.Left -
        msi.ActualWidth / msi.ViewportWidth * msi.ViewportOrigin.X;
        compositeTransform.TranslateY = msi.Margin.Top -
        msi.ActualWidth / msi.ViewportWidth * msi.ViewportOrigin.Y;
        compositeTransform.ScaleX = 1 / msi.ViewportWidth;
        compositeTransform.ScaleY = 1 / msi.ViewportWidth;
        return compositeTransform;
    }
}

private Point TransformGeometryPoint(Point pt)
{
    CompositeTransform ct = MsiTransform;
    return new Point((pt.X - ct.TranslateX) / ct.ScaleX, 
		(pt.Y - ct.TranslateY) / ct.ScaleY);
}

private double TransformGeometryLineLength(double len)
{
    return len * msi.ViewportWidth;
}

Support of the mark out feature also implies possibility to save (serialize) and than restore (deserialize) mark out information. Type MsiHelper is responsible for this. The mark out-related data are stored to Azure Blob Storage in XML format. Serialization is carried out using several auxiliary types, namely, PathTextualInfo, GeometryInfo, Saved and ItemToSave. And the following code shows geometric transform for deserialization of previously stored mark out data:

public void ProcessLoadedEditInfo(string xmlEditInfo)
{
    if (!string.IsNullOrEmpty(xmlEditInfo))
    {
        XmlSerializer serializer = new XmlSerializer(typeof(Saved));
        MemoryStream ms = new MemoryStream(Encoding.UTF8.GetBytes(xmlEditInfo));
        saved = serializer.Deserialize(ms) as Saved;
        ms.Close();

        allowChange = true;

        restoreActualWidthFactor = msi.ActualWidth / saved.msiActualWidth;

        msi.ViewportWidth = saved.msiViewportWidth * restoreActualWidthFactor;
        msi.ViewportOrigin = saved.msiViewportOrigin;
        msi.Margin = saved.msiMargin;
    }
}

Code Sample

Code for this article may be tested in local Azure development environment. You should start VS2010 "as Administrator", load AzureDz.sln solution to it, build and run AzureDz project. To upload an image file, you should navigate your browser to http://127.0.0.1:81/ImagingSvc.svc/Init and from that page, perform image file upload. To view uploaded image, navigate browser to http://127.0.0.1:81/SlImage.html. To operate the solution in Azure local development, environment values of StorageAccountName and StorageAccountKey parameters in file ServiceConfiguration.cscfg should be left empty.

To operate the sample from Azure cloud, you should first create a Storage Account and assign its name and primary access key to StorageAccountName and StorageAccountKey parameters in file ServiceConfiguration.cscfg. Then you should create a Hosted Service place-holder on the cloud, build release version, publish it and deploy to the newly created Hosted Service. Image upload can be performed from http://place-holder.cloudapp.net/ImagingSvc.svc/Init page. Image can be viewed navigating to http://place-holder.cloudapp.net/SlImage.html .

The workflow looks as following. Client sharing an image, navigates his/her browser to http://.../ImagingSvc.svc/Init on any browser-equipped device. Method Init() of ImagingSvc RESTful WCF service is called and returns a simple HTML page to browser in response. This HTML page allows client to upload an image labeling with a blob token (the image file name by default) with the WCF service. The image is uploaded as a stream by Upload() method of the ImagingSvc WCF service. This method also processes received image to a DZ image tile pyramid and puts it to Azure Blob Storage. Now the image can be viewed and marked out using any Silverlight-equipped browser. To view the image client should navigate his/her browser to http://.../SlImage.html and choose the image from Blob Token combo-box. The selected image is shown by MultiScaleImage Silverlight control. Its Source property is assigned (in Source property of MsiHelper) to the image pyramid XML blob residing in Azure Blob Storage.

User uploads image file operating HTML input tag of type "file" of his/her browser. It causes streamed HTTP POST request containing byte representation of image. This request is received by Upload() method of ImagingSvc WCF service. Sample of such request and its structure are shown in the table below:

The request is parsed in order to extract byte array for image as well as blob token and data type. Currently this parsing is carried out "manually" with methods of StaticHttpParser class (I am almost sure that there is a standard way with ready available types to parse this request, but I failed to find it quickly. Perhaps Upload() method should use parameter of System.ServiceModel.Channels.Message type...).

Our Silverlight-based client application supports mark out possibilities. User may switch GUI to edit mode by checking Edit checkbox (depicted below).

In edit mode, mouse operations cause mark out rather than image movement or zooming. Simple control panel above image allows user to choose mark out line thickness, line and fill colors and opacity parameters. Than, by pressing left mouse button and dragging mouse user defines selected rectangular mark area. On selection completion Highlighted Area Description dialog window (of type DescriptionChildWindow) appears prompting user to define layer, title and description for the selected area. Alternatively, by pressing Cancel button, user can cancel mark out of the latest selected area. Left mouse click inside selected area(s) in edit mode deletes the selection. Illustrating edit result, three selected objects are shown in the pictures. The objects are categorized to two layers, Characters (Alice, orange boundary, transparent filling, and White Rabbit, green boundary, red filling) and Things (White Rabbit's Watch, blue boundary, white filling). The objects have different degrees of opacity. After editing, client may be switched back to viewing mode by uncheck Edit checkbox. In viewing mode, currently selected mark out is highlighted with "blinking". Mark out may be toggled on / off by check / uncheck checkbox in Visible column of DataGrid object.

Discussion

Code presented in this article was tested on real Azure cloud with the largest image of 14 MB. After upload, appropriate message appeared in the browser (sometimes for large images, you may get some abnormal HTML response after upload, but normally upload was successful). In local development environment for large image file OutOfMemory exception may occur while preparing the image pyramid. The source of exception is put in the try-catch block, and appropriate after-upload message informs user that not all image layers will be viewed properly. As for mark out editing, currently appropriate features are quite limited. So there is room for further enhancements of algorithm improvements and editing features.

As it was stated above, usage of DZ technology required considerably larger storage space than simply storing of original image. If storage space is an issue, then the DZ image pyramid may be generated for a stored image ad hoc, or some caching mechanism (may be combining with appropriate scheduling) should be employed. For example, if it is known that tomorrow a doctor will examine X-ray images of certain patients then DZ image pyramid may be generated for these images during the night before the examination and destroyed after the examination.

Please note that code accompanied this article is not a complete product, but rather illustration of concept. So, for the sake of simplicity, some features important in the real world were intentionally omitted. These features are:

• Permissions for upload, viewing and annotating of images
• Support for different mark outs for different user or users group
• Policy for simultaneous mark out of the same image by several users
• Z-order of marked areas, etc., list is not exhaustive

Conclusions

This article presents a tool to upload image file to Azure cloud, process the image to a Deep Zoom image tile pyramid, store the pyramid images as blobs to Azure Blob Storage, view the image with MultiScaleImage control provided by Microsoft Silverlight, mark out and annotate areas of interest in the image. All the above operations are performed with just browser without any additional installations on user machine (the viewing browser however should support Silverlight).

References

[ 1] Deep Zoom. Wikipedia.
[ 2] Daniel Gasienica. Inside Deep Zoom – Part I: Multiscale Imaging.
[ 3] Daniel Gasienica. Inside Deep Zoom – Part II: Mathematical Analysis.
[ 4] Building a Silverlight Deep Zoom Application.
[ 5] Jaime Rodriguez. A deepzoom primer (explained and coded)..
[ 6] Sacha Barber. DeepZoom. CodeProject.
[ 7] Microsoft Deep Zoom Composer.
[ 8] Berend Engelbrecht. Generate Silverlight 2 DeepZoom Image Collection from Multi-page TIFF. CodeProject.
[ 9] Joerg Lang. Silverlight Database Deep Zoom. CodeProject.
[10] How can I put my Deep Zoom Image in Azure?

History

The initial version supported upload image from local machine / device to Azure Blob Storage and instant viewing of the stored image with possibility to scale, size and move. In the second version mark out, editing and serialization of appropriate data were added.