Note: Requires .NET 3.5 SP1 to run and the Multi-Touch Vista input service to enable multi-touch.
Contents
Introduction
Multi-touch has officially become a buzzword. Inspired by Minority Report, Microsoft Surface, the iPhone, and the new Windows 7 Touch APIs, developers have many reasons to invest in this trend. In this article, I'll start with an introduction to multi-touch development with WPF, with special focus in setting up the development PC and inputting or simulating multi-touch. Then, I'll show a touch RSS reader, along with some quirks and interesting points when developing for touch.
Multi-touch development by itself is not hard; the hardest part is correctly choosing your API and setting up your device. So, we'll start by...
Choosing the multi-touch API
To develop multi-touch apps with WPF, we have currently three main options:
The first solution is the one that aligns with Microsoft's current strategy. However, the WPF 4.0 beta 2 APIs are still in a very raw state; even though they are functional, they don't provide easy to use controls - you'll have to program most of the interaction logic by hand. Besides, you are limited to Windows 7 as deployment platform. This solution will certainly evolve, and in the future will probably become the best, but for now, we'll pass.
Surface SDK is much more interesting. It's a very complete API, with interesting controls and a growing community. Its main limitation is the hardware - it only works with MS Surface or its own simulator app.
Therefore, we'll go with the third solution. It's not only the most flexible solution (works with almost any hardware and OS), but it's also easy-to-use and Open Source. In this article, I'll use Multi-Touch Vista [^], a very complete package by Daniel D, a.k.a. "Nesher". It includes a flexible input system and a WPF framework with many controls, licensed under the GPL. Because of that, you'll have to publish your source code along with your apps, under the same license. In this article, I'll refer to Multi-Touch Vista as MTVista.
Multi-Touch Vista overview and basic set-up
The next step is to understand MTVista's input layer in order to correctly configure it. This is MTVista's basic architecture:
As you can see, if you don't want to write a custom input provider for MTVista, you'll need either a TUIO-compatible device or multiple mice for simulation. TUIO is a widely-adopted open protocol to broadcast touch messages. Most multi-touch hardware isn't TUIO-compatible out of the box, so here we have several cases:
- You have a Windows 7-compatible device, such as a Dell XT or HP TouchSmart:
- You have a Wiimote and infrared pens:
- You have another multi-touch device, such as a multi-touch table using CCV/TouchLib, a jailbroken iPhone, or an MS Surface:
- You don't have any device:
- Connect multiple mice to your PC and use MTVista's Multiple Mice provider to simulate multi-touch.
After you have a TUIO device or multiple mice correctly set up, you can download and extract the MTVista binaries or build them from source. I recommend building it from source so that you have the most recent code.
Both the source code and the demo project included with this article contain a copy of the Multi-Touch Vista binaries built from change set 29484.
All set? So, let's continue...
Starting up the Multi-Touch Vista input service
To start the MTVista input service, you can either:
- Run it directly each time, by running MultiTouch.Service.Console.exe, or
- Install MultiTouch.Service.exe as a Windows Service using installutil.exe (i.e., run %windir%\Microsoft.NET\Framework\v2.0.50727\installutil.exe /i Multitouch.Service.exe) and then set up automatic startup using services.msc.
After it's up and running, you should see one or more red dots in your screen (one for each connected mouse). That happens because MTVista uses the Multiple Mice provider as its default. If you want to use TUIO, you must run MultiTouch.Configuration.WPF.exe, select the TUIO provider, and click the blue arrow to apply your selection. You may need to unblock the service in Windows Firewall, as it broadcasts the TUIO messages locally using UDP packets.
That's it! The hardest part is done :-).
Developing a multi-touch app
To create a new application with MTVista, the path is almost always the same:
- Create a normal WPF application.
- Add a reference to MTVista's Multitouch.Framework.WPF library.
- On the XAML files, add reference to the MTVista namespace:
xmlns:mt="http://schemas.multitouch.com/Multitouch/2008/04"
- Change the
<Window> tag to <mt:Window> and change the code-behind class from System.Windows.Window to Multitouch.Framework.WPF.Controls.Window.
That's it! Now you have an empty multi-touch-enabled window:
The rest of the development flows the same way as a normal WPF app. In the next section, I'll show the demo app, detailing some interesting points that are specific to multi-touch development and some patterns that might be useful when working with multi-touch.
The demo application: a multi-touch RSS reader
RSS readers are everywhere today; they're simple, useful, and familiar - almost every reader looks the same: a list of feeds with or without categories, a list of news items for each feed, and a reading area for the selected article. But how can we make it more interesting and easy-to-use in a multi-touch environment?
The demo application here tries to change the common RSS reader layout a little bit. Its main features are:
- Integration with the Windows RSS platform (Internet Explorer Feeds) for getting its data.
- A panel-based system for folders, feeds, and items. The panels can be dragged and zoomed with multi-touch gestures, and can bounce off the "walls" of the application.
- A "trash" icon to close panels by dragging into it.
- Feeds can be read inside the application, without the need for an external browser.
Now, let's dive into each of these features do discover what's behind them.
Integration with the Windows RSS platform
To enable integration with the RSS platform, I've used Cristian Civera, a.k.a "Ricciolo"'s excellent PaperBoy library, part of his PaperBoy application [^]. This library wraps the RSS platform in a simple model, and exposes the data through a singleton called FeedManager.
In this app, we'll use the Model-View-ViewModel (MVVM) pattern to connect this model with our multi-touch front-end. MVVM is a pattern first described by John Gossman, and it's widely used to develop WPF applications. If you want to know more about it, take a look at Josh Smith's article on MSDN Magazine [^] or several others here in CodeProject [^]. In this project, I've used a simple set of ViewModels:
These ViewModels inherit from ObservableObject, from the MvvmFoundation Framework [^], developed by Josh Smith. This framework simplifies repetitive tasks with MVVM, such as inheriting from INotifyPropertyChanged and managing the PropertyChanged event.
Each of these ViewModels correspond to one data type we want to represent (Folders, Feeds, and Feed Items). They expose all of the properties that we want to display, like items, titles, and unread items, which come from the Model (Windows RSS Platform).
To initialize the application, a FolderViewModel pointing to FeedManager.Current.Root is created and bound to the corresponding view (more about this in the next section).
Multi-touch panel-based views
After the Model and ViewModels are set, we must think about the user interface. Users expect panels to have support for dragging, zooming, and rotating. Also, it's necessary to enable menus to work like an iPhone menu (with finger-scrolling and inertia).
To create a scrollable view using MTVista, all we have to do is wrap an ItemsControl in a MTVista ScrollViewer. Here, we'll use MTVista's DraggableScrollViewer because it also enables dragging support (it fires an event when you drag elements out of it). A scrollable view in this app without styles and behaviors would look like this:
[FolderAndFeedView.xaml] (modified):
<Border x:Class="VirtualDreams.TouchReader.View.FolderAndFeedView"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:mt="http://schemas.multitouch.com/Multitouch/2008/04">
<DockPanel>
<TextBlock DockPanel.Dock="Top"
Text="{Binding Title}"/>
<Rectangle DockPanel.Dock="Bottom" />
<mt:DraggableScrollViewer x:Name="scroll">
<ItemsControl ItemsSource="{Binding Items}">
<ItemsControl.ItemsPanel>
<ItemsPanelTemplate>
<VirtualizingStackPanel
Width="{Binding ElementName=scroll, Path=ActualWidth}" />
</ItemsPanelTemplate>
</ItemsControl.ItemsPanel>
</ItemsControl>
</mt:DraggableScrollViewer>
</DockPanel>
</Border>
Some interesting points about this view:
- The view is bound to a
ReadOnlyCollection<T> in the ViewModel (in this case, Items).
- The views are
Borders instead of UserControls because of a bug in the MTVista ScrollViewer with UserControls (see this [^] for details).
- We use a
VirtualizingStackPanel to enable better performance when we have many items to display.
The other views in the application are similar to this one. Next, we have to display our views on screen and enable multi-touch interactions (zooming, dragging, and scaling).
MTVista's TouchablePanel does the trick here - every child of it will support these natural gestures, in a similar way to the Surface SDK's ScatterView. MainWindow.xaml would look like this:
[MainWindow.xaml] (modified):
<mt:Window x:Class="VirtualDreams.TouchReader.MainWindow"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:mt="http://schemas.multitouch.com/Multitouch/2008/04"
xmlns:local="clr-namespace:VirtualDreams.TouchReader"
Title="TouchReader"
TextElement.FontFamily="Segoe, Calibri, Segoe UI"
TextElement.FontStretch="Condensed">
<Grid>
<mt:TouchablePanel x:Name="mainPanel"
AngularDamping="0.1"
LinearDamping="0.9"
EnableWalls="True"
RandomizePositions="False">
</mt:TouchablePanel>
<local:Trash HorizontalAlignment="Right"
VerticalAlignment="Bottom"
Margin="15" />
</Grid>
</mt:Window>
As you can see, there's nothing special about it:
- In the
TouchablePanel, the damping properties are used to customize inertia and friction. EnableWalls will make elements bounce off, and RandomizePositions is set to false to make sure we can control the initial positions of this panel's children.
- The
Trash element will be explained in detail in the next section.
Now we have a TouchablePanel, a set of Views and ViewModels. The next step is to enable an item to be dragged out of a view and create another view, depending on what's being dragged.
To do that, the main idea is to create a Factory-like class that would enable the TouchablePanel to create its children views. One solution would be to subclass the TouchablePanel and add methods to create the views, making it work as a Factory.
However, for this article, I've tried to leave this solution as general as possible: I wanted to create a reusable Factory interface that could be implemented in each specific case, allowing for more reusability. In this app, the Factory architecture looks like this:
The implementation of the Factory must decide how to create a view based on the DataContext of the new item and its position. It also needs to know the target Panel to add this new view to its children accordingly.
In this app, the implemented Factory method works like this:
- First, it checks if the DataContext is a valid
RssViewModelBase.
- If it is, it finds the correct view from its type.
- Then, it creates the view and associates it to the DataContext.
- The view is positioned using
Canvas.SetTop and Canvas.SetLeft. This works because the TouchablePanel is derived from Canvas.
- Next, it adds an event handler to this new view, enabling the element to capture contacts when touched. This enables views to be dragged without having to start the movement directly over them.
- Finally, it adds the view to the panel's children, thus displaying the view.
Finally, to connect the Factory to the views, we'll use Attached Behaviors (see this article from Josh Smith [^] for more). In this app, two behaviors are used:
- A behavior that allows us to link an
IPanelViewFactory to a Panel, and
- A behavior that handles the
DraggableScrollViewer's ElementDragged event and calls a Factory to create a View from the dragged element's DataContext.
The usages for these behaviors would look like this:
<Panel b:PanelBehavior.ViewFactory="{Binding [reference to the factory]}" ... />
<mt:DraggableScrollViewer
b:ElementDraggedCreateViewBehavior.PanelViewFactory=
"{Binding [reference to the factory]}" ... />
You can see the usages of these behaviors by taking a look at the source code in MainWindow.xaml and in the scrollable views (FolderAndFeedView.xaml).
This approach is very flexible: whenever you want to build another app that needs to control how views are created in a panel, all you have to do is implement an IPanelViewFactory and attach it in a similar way.
After everything is connected, the last thing is to initialize the application by calling the main panel's Factory to create the root folder view (from FeedsManager.Current.RootFolder):
[MainWindow.xaml.cs]
public MainWindow()
{
InitializeComponent();
PanelBehavior.GetViewFactory(mainPanel).CreateViewFromDataContext(
new FolderViewModel(FeedsManager.Current.RootFolder),
new Point(100, 100)
);
}
Trash icon to close views
The next feature we want to implement is a Trash icon. The idea here is to enable views to be dragged to the trash when we want to close them, and then hide them with an animation. Here, you can see a view being "swallowed" by the trash:
In this application, the Trash icon is just a simple UserControl:
[Trash.xaml]
<UserControl x:Class="VirtualDreams.TouchReader.Trash"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
Height="200"
Width="200">
<UserControl.Background>
<ImageBrush ImageSource="/Resources/Images/trash.png"
Opacity="0.8" />
</UserControl.Background>
</UserControl>
Now, let's break down the requirements for the Trash to find out what we need to make it work:
- The panels that will be dragged must be inside a Panel that supports dragging (usually a
TouchablePanel).
- In MTVista, the event that says that a contact has been "released" from an element is the
ContactRemoved event. In that case, the PreviewContactRemoved event will bubble up to the parent Panel.
- The Trash doesn't have to belong to the Panel - if it's contained in the panel, it will be "draggable", and that's not what we want here.
- A single Panel could have multiple trashes (e.g., a multi-user table where each user has his/her own trash), but a trash can only be associated to one Panel.
- The root folder can't be closed, or we will end up with an empty panel with nothing to add new items.
- When an element is dragged to the Trash, we want to use an animation to shrink it to zero pixels, giving the impression that the trash "swallows" the element.
As we can see from the first and third requirements, it would be useful if the Trash knew its source panel, so we can handle the panel's PreviewContactRemoved event and react accordingly.
In this app, I've used another Attached Behavior to add a SourcePanel property to the Trash control. The usage of this behavior looks like this:
<local:Trash b:TrashBehavior.SourcePanel="{Binding [reference to the panel]}" ... />
This behavior, when attached to a Trash control, will add it to a list of trashes associated with the panel. It also handles the panel's PreviewContactRemoved and attaches it to a handler that works like this:
- First, we check the dragged element to see if it's not the root folder or another trash.
- Then, we calculate the intersection between the element and each of this panel's trashes to see if there's overlapping.
- If an item overlaps a trash, an animation will be started. This animation acts on the
ScaleX, ScaleY, and Opacity properties, reducing them to zero. A dictionary is used to associate the animation to the dragged element.
- Finally, when the animation is over (the
CurrentStateInvalidated event fires), we use that dictionary to recover the element and remove it from the Panel.
For more details about this behavior, see the PreviewContactRemoved handler method in TrashBehavior.cs.
Reading feed items inside the application, without an external browser
The last feature in this demo app is the ability to read the feed contents in the multi-touch environment, without the need for an external browser. To do this, I've used another set of classes from Ricciolo's PaperBoy [^] application to convert the HTML code to a XAML FlowDocument. This set of classes is an evolution of a Microsoft sample that shipped with the WPF SDK (see this blog post [^] for more details).
To use this converter, all you need to do is use code similar to this:
xamlString = HtmlToXamlConverter.ConvertHtmlToXaml(htmlString, true);
In this method, call:
- The first parameter is the HTML content as a string
- The second parameter is a boolean that indicates if the content should be wrapped in a
FlowDocument
- The return value is the XAML string converted from the HTML source
After the conversion, all we need to do is to parse the XAML code using a XamlReader:
FlowDocument document = XamlReader.Parse(xamlString) as FlowDocument;
In this demo app, the FlowDocument is displayed in a FlowDocumentScrollViewer inside a MTVista ScrollViewer, to enable touch panning:
[FeedItemView.xaml]
<mt:ScrollViewer VerticalScrollBarVisibility="Hidden">
<FlowDocumentScrollViewer Foreground="White"
ScrollViewer.VerticalScrollBarVisibility="Hidden"
Document="{Binding Content}" />
</mt:ScrollViewer>
For more details about the HTML to XAML conversion and display, see FeedItemViewModel.cs and FeedItemView.xaml. I'd also recommend a visit to the WPF SDK sample [^] and to the HTMLConverter classes included with the source code.
Wrapping up
In this article, we touched the fundamental aspects of multi-touch development with WPF, like set up and choice of the API. We've seen a demo app that contains many common patterns that you may find interesting in your multi-touch apps, such as the "Trash" and creating views from dragged elements.
I hope that this article could provide you with some tools and ideas to start your own multi-touch app development. I'm looking forward to your next-generation multi-touch experiences!
For the future
Some interesting next steps for this app could be:
- Developing a version with the WPF 4.0 Beta 2 APIs or with the Surface SDK.
- Using Chris Cavanagh's excellent WPF Chromium WebBrowser [^] to implement a full-featured browser inside the application.
- Implementing a "shelf" system to store favorite items (e.g., Tafiti [^]). The shelf implementation is similar to the Trash.
- Implementing a "Recycle Bin" feature, enabling users to see what's in the trash and restore items.
- Creating user interfaces for subscribing/unsubscribing to feeds instead of using the Windows RSS platform.
- Implementing database or web storage (e.g., Google Reader synchronization).
What do you think?
Was this project useful? What do you think of the architecture patterns presented?
Please leave your comments and suggestions, and please vote up if this article pleases you. Thanks!
Other links and references
History
- v1.0 (11/01/2009) - Initial release.
