Introduction
Interaction Context is a new Windows 8 API designed for developers who are building UI frameworks that provide a consistent touch-optimized user experience across desktop applications. This article introduces a managed wrapper for this API and an example of using Interaction Context in a WinForms application.
Background
If you’re planning to support touch input in a desktop application, you have a few options:
Processing WM_GESTURE
message is the simplest option. It is compatible with Windows 7 and supports useful gestures, such as panning, zooming, and rotation. While panning, you can limit perpendicular movement to primary direction (use the gutter) and use inertia to smoothly slow when the pan gesture stops. It supports the two-finger tap and the press and tap gestures. There is no a dedicated Tap gesture however. You have to wait for the WM_LBUTTONUP
message and distinguish between the mouse and touch input using the GetMessageExtraInfo()
function from Win32 API. The main problem of the WM_GESTURE
message is that it doesn’t support multi-touch interactions. For example, you can’t start moving a tile with one hand and scroll the list of tiles with another hand, as on Windows 8 Start Screen. Also, you can’t perform a complex manipulation, such as pan, zoom, and rotation at the same time.
WM_TOUCH
message is a more advanced thing. It is also compatible with Windows 7 and offers the ability to work with multiple touch inputs at once. It doesn’t detect any high-level gestures however. You just obtain an array of (x, y) points and flags indicating whether the specified touch point was added, moved, or removed. That may be enough for some applications, such as the piano simulator or something like that. For other applications, that’s not enough and you want to detect touch gestures. If you’re programming in C++, you can use the standard IManipulationProcessor
and IInertiaProcessor
COM interfaces. They are pretty powerful but require implementation of event sinks. I’m not aware whether anybody tried to work with those interfaces from managed code. Perhaps, it’s more efficient to create the managed counterparts, such as the classes from System.Windows.Input.Manipulations
namespace in .NET 4. In any case, processing WM_TOUCH
messages is tricky and error-prone. They are also followed with fake mouse messages that you need to distinguish from real mouse messages.
Windows 8 introduces the concept of Pointer Input Messages. These messages come from various input devices, such as touch, pen, or mouse (after executing EnableMouseInPointer
function). They can also be “injected” programmatically with the new InjectTouchInput
function. The WM_POINTER*
messages are somewhat similar to WM_TOUCH
but give a lot more information about user input. You can easily distinguish mouse input from touch input, for example. The special WM_TOUCHHITTESTING
message helps to determine the most probable touch target and so on. Of course, it supports multi-touch input. But, in general, as with WM_TOUCH
message, you have only (x, y) coordinates of the touch point (or mouse click) and the information about whether the pointer was added, moved, or removed. WM_POINTER*
messages are low-level stuff and don’t tell us anything about touch gestures.
While browsing through MSDN, I discovered a section about the new API that enables Windows 8 applications to support multiple, concurrent interactions by providing gesture detection and manipulation processing. This is about Interaction Context. Its documentation is still very poor and foggy. I was unable to use it until I’ve found an excellent article and sample on Intel’s web site.
Interaction Context is an API that works in conjunction with WM_POINTER*
messages. It accepts low-level data from pointers then executes a callback function passing high-level information about touch interactions as an argument to that function. Using Interaction Context functions is much easier than handling IManipulationProcessor
and IInertiaProcessor
interfaces in case of WM_TOUCH
message. You don’t need to bother with COM and event sinks anymore. It is even possible to create a managed wrapper for this API and use it from C# code.
Flexibility and simplicity are the main advantages of the new touch API. It separates touch destinations from windows. For example, in the InteractionContextTest
sample, you can see a few geometric figures. They are not windows, just objects that can draw themselves on the Form. Each of these figures and the background creates and adjusts its own interaction context and, therefore, can show various behaviors when you move figures with the mouse or touch device. These contexts are independent of each other. So you can, for example, move two figures with two hands at the same time. And, even more, you can move/zoom/rotate one figure with the touch device while moving another figure with the mouse. That was almost impossible with the WM_TOUCH
message.
Also, the visual feedback is more consistent for the Interaction Context. For example, if you enabled the “press&hold” behavior for some context (that simulates right mouse click), the first pointer added to this context will display a square on the screen after some time. The pointers that don’t belong to this context will not display a square despite the fact that all the pointers are passed to the same window. This means the interaction context may affect the associated pointers, not just get data from pointers.
Creating Interaction Context
In the sample, I moved all interop code to the Win32.cs file. There you can find definitions of many enumerations and structures that are passed to/from the API functions. These functions are also defined there as DllImports
. See MSDN for more information about Pointer Input Messages and Interaction Context.
The object that needs to be associated with an interaction context must implement the IInteractionHandler
interface. It is very simple:
internal interface IInteractionHandler
{
void ProcessInteractionEvent(InteractionOutput output);
}
The InteractionOutput
class contains data passed from the interaction context to a callback function:
internal class InteractionOutput
{
internal Win32.INTERACTION_CONTEXT_OUTPUT Data;
internal IntPtr InteractionContext { get; }
internal bool IsBegin();
internal bool IsInertia();
internal bool IsEnd();
}
Implementation of the IInteractionHandler
interface can be found in the sample within the BaseHandler
class. Both Figure
and Background
classes are derived from BaseHandler
. The callback function is shared between all interaction contexts. It is a part of the infrastructure in Win32
static class. You can see there some tricks that make it thread-safe and allow processing inertia for all interaction contexts with a single timer.
To create an interaction context, we must call the Win32.CreateInteractionContext
method. It accepts the IInteractionHandler
interface as a parameter and the current SynchronizationContext
:
IntPtr _context;
public BaseHandler()
{
_context = Win32.CreateInteractionContext(this, SynchronizationContext.Current);
}
In the object’s Dispose
method, we must call the Win32.DisposeInteractionContext
method:
public void Dispose()
{
Win32.DisposeInteractionContext(_context);
_context = IntPtr.Zero;
}
Before using the interaction context, it has to be configured. For example, in the constructor of the Background
class, we use the following code to update its configuration:
Win32.INTERACTION_CONTEXT_CONFIGURATION[] cfg = new Win32.INTERACTION_CONTEXT_CONFIGURATION[]
{
new Win32.INTERACTION_CONTEXT_CONFIGURATION(Win32.INTERACTION.TAP,
Win32.INTERACTION_CONFIGURATION_FLAGS.TAP |
Win32.INTERACTION_CONFIGURATION_FLAGS.TAP_DOUBLE),
new Win32.INTERACTION_CONTEXT_CONFIGURATION(Win32.INTERACTION.SECONDARY_TAP,
Win32.INTERACTION_CONFIGURATION_FLAGS.SECONDARY_TAP),
new Win32.INTERACTION_CONTEXT_CONFIGURATION(Win32.INTERACTION.HOLD,
Win32.INTERACTION_CONFIGURATION_FLAGS.HOLD)
};
Win32.SetInteractionConfigurationInteractionContext(Context, cfg.Length, cfg);
The background doesn’t support any interactions, except tap, double tap, and secondary tap (right click). We also enable the hold interaction to give better feedback for the secondary tap interaction.
In the constructor of the Figure
class, we enable various manipulations depending on the parameters passed to the constructor:
Win32.INTERACTION_CONTEXT_CONFIGURATION[] cfg = new Win32.INTERACTION_CONTEXT_CONFIGURATION[]
{
new Win32.INTERACTION_CONTEXT_CONFIGURATION(Win32.INTERACTION.MANIPULATION,
Win32.INTERACTION_CONFIGURATION_FLAGS.MANIPULATION |
Win32.INTERACTION_CONFIGURATION_FLAGS.MANIPULATION_SCALING |
Win32.INTERACTION_CONFIGURATION_FLAGS.MANIPULATION_ROTATION |
Win32.INTERACTION_CONFIGURATION_FLAGS.MANIPULATION_TRANSLATION_INERTIA |
Win32.INTERACTION_CONFIGURATION_FLAGS.MANIPULATION_ROTATION_INERTIA |
Win32.INTERACTION_CONFIGURATION_FLAGS.MANIPULATION_SCALING_INERTIA),
new Win32.INTERACTION_CONTEXT_CONFIGURATION(Win32.INTERACTION.TAP,
Win32.INTERACTION_CONFIGURATION_FLAGS.TAP |
Win32.INTERACTION_CONFIGURATION_FLAGS.TAP_DOUBLE)
};
if (!pivot)
{
cfg[0].Enable |=
Win32.INTERACTION_CONFIGURATION_FLAGS.MANIPULATION_TRANSLATION_X |
Win32.INTERACTION_CONFIGURATION_FLAGS.MANIPULATION_TRANSLATION_Y;
}
if (rails)
{
cfg[0].Enable |=
Win32.INTERACTION_CONFIGURATION_FLAGS.MANIPULATION_RAILS_X |
Win32.INTERACTION_CONFIGURATION_FLAGS.MANIPULATION_RAILS_Y;
}
Win32.SetInteractionConfigurationInteractionContext(Context, cfg.Length, cfg);
Configuration parameters can be assigned at any time but the changes are applied only to new interactions.
Processing Pointer Messages
In the Form’s WndProc
method, we should handle these messages:
WM_POINTERDOWN
WM_POINTERUP
WM_POINTERUPDATE
WM_POINTERCAPTURECHANGED
To get the pointer ID
and its POINTER_INFO
from window message parameters, we use the following code:
int pointerID = Win32.GET_POINTER_ID(m.WParam);
Win32.POINTER_INFO pi = new Win32.POINTER_INFO();
if (!Win32.GetPointerInfo(pointerID, ref pi))
{
Win32.CheckLastError();
}
WM_POINTERDOWN
message occurs when a user touches his screen or presses the mouse button and, hence, a new pointer is added. We need to do hit-testing when processing WM_POINTERDOWN
message and determine a figure that contains the new pointer. Then we should add the pointer to the figure’s interaction context using the AddPointerInteractionContext
function. All subsequent messages from this pointer can be passed to the same context without hit-testing because the pointer ID
can easily be found in the figure’s ActivePointers
hash-set.
The following is a simplified version of code processing WM_POINTERDOWN
message:
case Win32.WM_POINTERDOWN:
{
Point pt = PointToClient(pi.PtPixelLocation.ToPoint());
foreach (Figure f in _figures)
{
if (f.HitTest(pt.X, pt.Y))
{
f.AddPointer(pointerID);
f.ProcessPointerFrames(pointerID, pi.FrameID);
break;
}
}
}
break;
The HitTest
method in the Figure
class determines whether the point is contained within the bounds of the given figure. The method is simple still powerful. It can be used for any convex figures, not just rectangles. The AddPointer
method associates the pointer ID
with an interaction context and adds the ID
to the ActivePointers
hash-set of the given figure. ProcessPointerFrames
method’s code is the following:
int _lastFrameID = -1;
public void ProcessPointerFrames(int pointerID, int frameID)
{
if (_lastFrameID != frameID)
{
_lastFrameID = frameID;
int entriesCount = 0;
int pointerCount = 0;
if (!Win32.GetPointerFrameInfoHistory(pointerID, ref entriesCount,
ref pointerCount, IntPtr.Zero))
{
Win32.CheckLastError();
}
Win32.POINTER_INFO[] piArr = new Win32.POINTER_INFO[entriesCount * pointerCount];
if (!Win32.GetPointerFrameInfoHistory(pointerID, ref entriesCount, ref pointerCount, piArr))
{
Win32.CheckLastError();
}
IntPtr hr = Win32.ProcessPointerFramesInteractionContext(_context,
entriesCount, pointerCount, piArr);
if (Win32.FAILED(hr))
{
Debug.WriteLine("ProcessPointerFrames failed: " + Win32.GetMessageForHR(hr));
}
}
}
This method obtains the entire frame of information for the specified pointer and passes this frame to the interaction context.
WM_POINTERUP
message occurs when a pointer is removed. We have to remove the pointer from ActivePointers
hash-set and from the interaction context:
case Win32.WM_POINTERUP:
foreach (Figure f in _figures)
{
if (f.ActivePointers.Contains(pointerID))
{
f.ProcessPointerFrames(pointerID, pi.FrameID);
f.RemovePointer(pointerID);
break;
}
}
break;
WM_POINTERUPDATE
message usually indicates that the pointer location is changed. Handling is trivial:
case Win32.WM_POINTERUPDATE:
foreach (Figure f in _figures)
{
if (f.ActivePointers.Contains(pointerID))
{
f.ProcessPointerFrames(pointerID, pi.FrameID);
break;
}
}
break;
WM_POINTERCAPTURECHANGED
message may occur when the window is losing capture. If so, we remove all pointers and stop the interaction context.
Processing Interaction Events
Eventually, the interaction context’s callback function executes the ProcessEvent
method in the classes derived from BaseHandler
. Use the IsBegin()
, IsEnd()
, and IsInertia()
methods of the InteractionOutput
object to determine the state of interaction processing. You should take a look at the definition of the INTERACTION_CONTEXT_OUTPUT
structure that is passed to the ProcessEvent
method as output.Data
. The type of current interaction can be taken from the Interaction
field (such as manipulation, tap, cross-slide, and others). The source of interaction can be obtained from InputType
(such as mouse, pen, or touch). There are also the special sub-structures for manipulation, tap, and cross-slide interactions. For example, it is used in Background
's code:
internal override void ProcessEvent(InteractionOutput output)
{
if (output.Data.Interaction == Win32.INTERACTION.TAP)
{
if (output.Data.Tap.Count == 2)
{
foreach (Figure f in _form.Figures)
{
f.ResetPoints();
}
_form.Invalidate();
}
}
}
In the InteractionContextTest
sample, you can double-click the background to reset the position of all the figures. Also, the press&hold gesture (or mouse right-click) changes the background color. Figures can be moved (except the pinned one), scaled, rotated, and double-clicked. The pinned figure can be rotated with one finger. Hopefully, you’ll find this sample useful.
Conclusion
Windows 8 introduces the new APIs for unified processing input from various positioning devices. Pointer Input Messages and Interaction Context functions are especially helpful when dealing with touch input. You can use the InteractionContextTest
sample as the base for your own development. Adding touch support to Windows 8 desktop applications becomes easy if you don't have to provide backward compatibility with Windows 7.