Handling Large Data Sets in Silverlight using WCF and Customized DataGrids

• Download source - 402.46 KB

Introduction

This article gives you a brief introduction in advanced techniques for up- and downloading large datasets and editing them. Therefore only standard libraries delivered with Silverlight 3 are used. The communication is based on WCF services and the UI only uses datagrids, buttons and textboxes. The interesting part of this article is that it shows you how to overcome some common obstacle that you easily can bump into by just using extension classes, reflection and some simple but powerful tricks.

Background

When I started using Silverlight, I took a brief look at some examples and built the first sample applications very fast. I had some small problems with data binding and XAML but after understanding the basic concepts, everything seemed to be easy. I found many very good examples covering the basic concepts which gave me a good start and made things look easy. Then I had to implement the first real application and things became problematic. What I wanted to implement seemed to not be very difficult - when you would implement it in WinForms - but with Silverlight, I soon found out that it would not work the way you would do it with WinForms/MFC. Here is what the application should do:

• Handle large amount of data: up- and download over 50,000 rows of data and more than 10 MB of data
• Display the data in a datagrid fast (high performance)
• Add data without using a pop-up dialog - just always have an empty line at the end
• Sort data
• Filter data

All this sounds simple but it is not. The first problem is that uploading more than 8 KB of data using WCF services does not work out of the box. The second problem is that adding an empty line to a datagrid is easy, but in combination with sorting the empty line is sorted as well and is not always at the end: if you change sort direction, it becomes the first row. And finally performance can become a problem. Sorting 50,000 rows takes several seconds when using an unmodified datagrid and this is definitely not satisfying, especially if you compare it to WinForms.

Overview of this Solution

Basically this solution consists of two parts:

• The WCF service for handling large data sets
• The extensions to the datagrid for displaying, editing, sorting and filtering the data

First we will take a look at the WCF service so that we have some data to play with. Then we will take a look at the datagrid. This demo uses Silverlight 3.0, .NET 3.5 and was written with Visual Studio 2008. It should run by just opening the solution and pressing F5. I will not go into detail on how to deploy it (client and server on different machines) or go too much into detail on how to define services, XAML files, etc. using Visual Studio 2008. There already exist a lot of very good examples - also here on CodeProject - so I think I could not make it better or add something to these examples.

The WCF Service

The WCF service is implemented server side and part of the "MappingDataEditor.Web" project. Loading large amounts of data works (almost) out of the box. In this example, the content of "test_data.csv" (located in the subdirectoy "TestData") is loaded 50 times. The file has 1,000 rows so 50,000 rows are loaded resulting in ca. 11 MB of data sent from the server to the client. In the "web.config", make sure to set "dataContractSerializer" parameter:

<dataContractSerializer maxItemsInObjectGraph="2147483647"/>

Uploading is a little bit more complicated because you cannot upload more than 8 KB with one call. So the data is transferred in blocks. Client side is managed by the UploadState-object. The methods needed on the server side look like this:

[OperationContract]
Guid GetHandle(string pTableame);

[OperationContract]
void AddChunk(Guid pHandle, string pBlock);

[OperationContract]
bool Complete(Guid pHandle);

The data is transferred using the following steps:

• The data in the table is serialized
• The UploadState-object is initialized and after calling StartUpload communication is handled completely by the UploadState-object
• The first block is sent to the server using the WCF call "GetHandle"
• Then the data is transferred using 8KB blocks with multiple calls to "PushOneBlock"
• After each block transfer the calling process is informed about the progress (you can register on the "ProgressChanged" event)
• After transferring the final block, the method "Complete" is called. Now the server side service knows that all data is transferred and can do whatever should be done with the data (in this case, saving data)

The Datagrid

The datagrid uses the SortCollectionView object which implements the interfaces ICollectionView and IList.

public class SortCollectionView<T> : ICollectionView, IList<T>
{
    private ObservableCollection<T> myContent = new ObservableCollection<T>();
    ...

Now sorting and filtering can be implemented by ourselves. This way we can make sure that an empty line for entering new data is always at the end. We just remove the last line, order the data and add the empty line again:

object selectedItem = myDGTable1.SelectedItem;
SortDescription sortDesc = myReturnTableView.SortDescriptions[0];
this.myReturnTableView.Content.RemoveAt(this.myReturnTableView.Content.Count - 1);
if (myChGeneric.IsChecked == true)
{
    myReturnTableView.Content.SortQuick(new GenericComparer<ReturnTableEntry>
	(sortDesc.PropertyName, sortDesc.Direction == ListSortDirection.Ascending, 
	StringComparison.CurrentCulture, myReturnTableView[0]));
}
else
{
    if (sortDesc.Direction == ListSortDirection.Ascending)
    {
        myReturnTableView.Content.SortQuick(new Field1ComparerUp());
    }
    else
    {
        myReturnTableView.Content.SortQuick(new Field1ComparerDown());
    }
}

myDGTable1.SelectedItem = null;
myReturnTableView.Content.Add(new ReturnTableEntry());
myReturnTableView.FireCollectionChanged();
myDGTable1.SelectedItem = selectedItem;

Also we can implement filtering by passing a FilterCallback function and applying it on the data:

public IEnumerator<T> GetEnumerator()
{
    if (myFilter == null)
    {
        return this.myContent.GetEnumerator();
    }
    else
    {
        Collection<T> col = new Collection();
        foreach (T item in this.myContent)
        {
            if (myFilter(item))
            {
                col.Add(item);
            }
        }

        return col.GetEnumerator();
    }
}

Using the Code

Now that we took a brief look at the interesting parts of this example application, here is a listing of the interesting files and what they are used for:

• MainPage.xaml: UI definition in XAML defining the layout of Tabpage, Canvas, Table, Buttons and Textboxes
• MainPage.xaml.cs: Code behind file containing most of the UI logic
• SortCollectionView.cs: Object for sorting and filtering an ObservableCollection
• ChangedSortDescriptionCollection.cs: Needed to get access to the "CollectionChanged" event
• Extensions.cs: Generic extension methods for sorting ObservableCollection (try out the Bubblesort and see how slow it is compared to Quicksort)
• UploadState.cs: Helper class for uploading large amounts of data as a string. Extending this one with a ZIP-compressor (a great implementation can be found here) could speed up the upload.
• IDataService.cs: Interface of the WCF service
• DataService.svc.cs: Server side implementation of the WCF service
• DataServiceHostFactory.cs: "ServiceHostFactory" for the service to prevent problems when you use multiple services
• Default.aspx: Website hosting the Silverlight application
• UploadManager.cs: Keeps track of the received block when uploading the data in multiple small blocks of data
• UploadSession.cs: Implementation of one upload session
• UploadSessions.cs: Cache for all currently active "UploadSession"s
• ReturnTableEntry.cs: Simple object used in this example for representing one line in the table

Points of Interest - Performance Considerations

There are some very interesting observations I made while evaluating performance. The sorting is based on a Quicksort algorithm. First I implemented a generic sorter which determines by using reflection which type the ordered column has:

this.myPropertyName = value;
this.myPropInfo = this.myTypeInstance.GetType().GetProperty(this.PropertyName);
if (this.myPropInfo.PropertyType.Equals(typeof(string)))
{
    this.CompareFunction = this.StringComparer;
}
else if (this.myPropInfo.PropertyType.Equals(typeof(int)))
{
    this.CompareFunction = this.IntComparer;
}
else if (...

Depending on the type, it sets the comparer that should be used. When sorting the data, the comparer gets the values using reflection again:

private int StringComparer(T pTypeInstance1, T pTypeInstance2)
{
    string string1 = (string)myPropInfo.GetValue(pTypeInstance1, null);
    string string2 = (string)myPropInfo.GetValue(pTypeInstance2, null);
    ...

On a Core2Duo with 2.7 GHz, it takes more than 8 seconds sorting the 50,000 rows in the table. But if you do not use reflection and pass a comparer which accesses the properties directly, the sorting takes less than 1 second making everything more than 10 times faster. So reflection is great when you want to implement a generic solution but it has a great impact on performance.

The second interesting thing is that the first time you execute some code, it always takes approximately 20% more time than the following calls. Here you can see the impact of the .NET interpreter on the code. You can try this out by taking a look at the lower part of the application. In the textbox, you can see how long the operations last.

Also take a look at the SortCollectionView. Internally it stores the data in an ObservableCollection. This way you can set the data directly as it comes from the WCF service without any conversion. So setting the data and displaying it only takes 0.1 second. If you would copy the data over into another format, it would take a few seconds.

Remarks

This is just an example application and you should not use it in a productive environment before testing it thoroughly. Also this implementation only works with ObservableCollections and Properties. It will not work with DataViews. Anyway I hope you like it, and that it saves you a lot of time. Have fun with it!

History

• 5^th April, 2010: Version 0.1 for Silverlight 3 of the test application "MappingDataEditor"
• 2^nd July, 2010: New version for Silverlight 4.0 and .NET 4.0
  • Removed a bug with the checkbox
  • Added functionality for paged loading
  • Remark: Performance not optimized for large datasets. Please take a look at TPL and parallel loops
  • The access path to the Testdata is hardcoded in the Web.config. Please edit it or extract the project folder directly to "C:\"
  • Pressing the "create"-Button generates a new, large TestData-File with 100000 lines. After this loading data in one Block could fail. Only Paged load works then. Adjust the number of lines that will be created in "DataService.svc.cs".