At PDC 2009, Reed Townsend presented some very exciting multitouch samples in a presentation: Windows Touch Deep Dive. The following video shows the presentation:
Many customers have been asking about how to properly handle multiple window objects and the first half of his presentation does a great job explaining the best practices. It's been a while since Reed presented this sample but I have the source code and did a short write up interpreting how everything is working. You can download the Windows Touch Photostrip sample from my server and the following notes explain the various pieces.
Summary
The PhotoStrip control created for this demo is similar to the one that is included in the Touch Pack and on the Microsoft surface. For this demo, not only is the control created, but an entire framework for having multiple reusable controls that can be mixed and placed into applications.
Typical goals for a control such as the demo PhotoStrip control could be to:
- Respond to touch messages
- Add constrained manipulations for panning
- Add support for dragging in / out of the control
- Add an AIP surface
- Flesh out the overlay and gallery
The following image shows the PhotoStrip control in the demo application.
There are PhotoStrip controls on the top and the bottom of the screen. Sliding along the control will move the control. Dragging images from the bottom of the control up will cause them to appear in the Gallery.
Demo Overview
The PhotoStrip control is encapsulated into its own HWND
. This is a problem if you want to have many controls working well together because Windows Touch messages are only sent to a single window, the first window that receives the touch down message. To work around this, an overlay window takes all the input and maps it to the correct control as appropriate based on hit detection. The following diagram outlines this hierarchy of Windows.
Given that window hierarchy with the overlay, the following shows how data would flow through the control.
In the message flow, a WM_TOUCH
message is generated by the input hardware. This message will be received by the overlay HWND
. The overlay HWND
then performs hit detection and sends the message to the correct child HWND
(the PhotoStrip window, for example). If the user were dragging an image to the gallery, the PhotoStrip would then send a message to the application which tells it that the user dragged a photo to the gallery. The application then tells the gallery to display the photo in the gallery.
The following image illustrates the controls of the demo application in a screenshot of the demo running.
There are PhotoStrip controls on the top and bottom of the application and the Gallery control is in the center of the application layout.
Programming Details
Handling Messaging to the Overlay Control
First off, the application creates the overlay window and then creates the controls for the gallery and the PhotoStrips. These controls are then registered with the overlay.
The following code shows how this is done in code.
HRESULT InitWindow( HINSTANCE hInstance )
{
(…)
if( SUCCEEDED( hr ) )
{
hWnd = CreateWindowEx(0, WINDOW_CLASS_NAME, L"CollageSample",
WS_OVERLAPPEDWINDOW, CW_USEDEFAULT, CW_USEDEFAULT,
CW_USEDEFAULT, CW_USEDEFAULT, NULL, NULL, hInstance, 0);
if (!(hWnd))
{
hr = E_FAIL;
}
}
if( SUCCEEDED( hr ) )
{
ShowWindow( hWnd, SW_SHOWMAXIMIZED );
DisableTabFeedback(hWnd);
}
if (SUCCEEDED( hr ))
{
GetClientRect((hWnd), &rClient);
iWidth = rClient.right - rClient.left;
iHeight = rClient.bottom - rClient.top;
iTopHeight = (INT)(iHeight * TOP_HEIGHT_PERCENTAGE);
iBottomHeight = (INT)(iHeight * BOTTOM_HEIGHT_PERCENTAGE);
hr = Overlay::CreateOverlay(hInstance, hWnd, 0, 0,
iWidth, iHeight, &g_pOverlay);
}
if (SUCCEEDED( hr ))
{
hr = Photostrip::CreatePhotostrip(hInstance, (hWnd),
0, 0,
iWidth, iTopHeight,
TOP_PHOTOSTRIP_DIRECTORY,
Photostrip::Top,
&g_pStripTop);
}
if (SUCCEEDED( hr ))
{
hr = Photostrip::CreatePhotostrip(hInstance, (hWnd),
0, rClient.bottom - iBottomHeight,
iWidth, iBottomHeight,
BOTTOM_PHOTOSTRIP_DIRECTORY,
Photostrip::Bottom,
&g_pStripBottom);
}
if (SUCCEEDED( hr ))
{
hr = Gallery::CreateGallery(hInstance, hWnd, 0, iTopHeight,
iWidth, (iHeight - iTopHeight - iBottomHeight), &g_pGallery);
}
if (SUCCEEDED( hr ))
{
g_pOverlay->RegisterTarget((ITouchTarget*)g_pGallery);
g_pOverlay->RegisterTarget((ITouchTarget*)g_pStripBottom);
g_pOverlay->RegisterTarget((ITouchTarget*)g_pStripTop);
}
When WM_TOUCH
messages are generated, the overlay window will receive them in its WndProc
method. The overlay converts the message coordinates from screen coordinates to client coordinates and keeps track of the messages. The overlay keeps track of particular inputs and then gives capture to a particular control.
The following code shows how the overlay maps inputs to the appropriate child control.
LRESULT CALLBACK Overlay::S_OverlayWndProc(HWND hWnd, UINT msg, WPARAM wParam,
LPARAM lParam)
{
LRESULT result = 0;
INT iNumContacts;
PTOUCHINPUT pInputs;
HTOUCHINPUT hInput;
POINT ptInputs;
ITouchTarget *pTarget = NULL;
Overlay* pOverlay;
pOverlay = (Overlay*)GetWindowLongPtr(hWnd, 0);
if (pOverlay != NULL)
{
switch(msg)
{
case WM_TOUCH:
iNumContacts = LOWORD(wParam);
hInput = (HTOUCHINPUT)lParam;
pInputs = new (std::nothrow) TOUCHINPUT[iNumContacts];
if(pInputs != NULL)
{
if(GetTouchInputInfo(hInput, iNumContacts, pInputs, sizeof(TOUCHINPUT)))
{
for(int i = 0; i < iNumContacts; i++)
{
if (pInputs[i].dwFlags & TOUCHEVENTF_DOWN)
{
pOverlay->m_ucContacts++;
}
else if (pInputs[i].dwFlags & TOUCHEVENTF_UP)
{
pOverlay->m_ucContacts--;
}
ptInputs.x = pInputs[i].x/100;
ptInputs.y = pInputs[i].y/100;
ScreenToClient(hWnd, &ptInputs);
pInputs[i].x = (LONG)(ptInputs.x);
pInputs[i].y = (LONG)(ptInputs.y);
if (pInputs[i].dwFlags & TOUCHEVENTF_DOWN)
{
pOverlay-> m_pDispatch->CaptureCursor(pInputs[i].dwID,
(FLOAT)pInputs[i].x, (FLOAT)pInputs[i].y, &pTarget);
}
if (pOverlay->m_pDispatch->GetCapturingTarget(pInputs[i].dwID, &pTarget))
{
pTarget->ProcessTouchInput(&pInputs[i]);
}
}
}
delete [] pInputs;
}
CloseTouchInputHandle(hInput);
break;
case WM_LBUTTONUP:
ReleaseCapture();
pOverlay->m_fIsMouseDown = FALSE;
pOverlay->ProcessMouse(msg, wParam, lParam);
break;
case WM_MOUSEMOVE:
pOverlay->ProcessMouse(msg, wParam, lParam);
break;
case WM_LBUTTONDOWN:
SetCapture(hWnd);
pOverlay->m_fIsMouseDown = TRUE;
pOverlay->ProcessMouse(msg, wParam, lParam);
break;
case WM_DESTROY:
SetWindowLongPtr(hWnd, 0, 0);
PostQuitMessage(0);
delete pOverlay;
break;
default:
result = DefWindowProc(hWnd, msg, wParam, lParam);
}
}
else
{
result = DefWindowProc(hWnd, msg, wParam, lParam);
}
return result;
}
Each of the controls handles the touch messages in a similar manner in the ProcessTouchInput
method which will cause _ManipulationEvents
to be raised on the control.
Handling Touch Input for the PhotoStrip
The PhotoStrip itself implements the InertiaObj
utility interface to enable inertia features. The following diagram illustrates the WM_TOUCH
input mapping from the overlay window to the PhotoStrip control and its photos.
When capture goes to the control, the WM_TOUCH
messages are sent to a ManipulationEvents interface
which will raise manipulation events on that interface
. Horizontal movement within the control will manipulate all of the photos simultaneously. Hit testing within the PhotoStrip
control is used to send input to the appropriate photo because the photos themselves can be manipulated vertically within the control. Once the PhotoStrip control has finished interpreting input, that control releases capture for the control.
The following code shows how input is mapped to manipulations on the ManipulationDelta
handler.
HRESULT STDMETHODCALLTYPE Photostrip::ManipulationDelta(
FLOAT ,
FLOAT ,
FLOAT translationDeltaX,
FLOAT ,
FLOAT ,
FLOAT ,
FLOAT ,
FLOAT ,
FLOAT ,
FLOAT ,
FLOAT ,
FLOAT )
{
HRESULT hr = S_OK;
FLOAT fpTotalWidth = -INTERNAL_MARGIN, fpNewXOffset;
std::list<Photo*>::iterator it;
for (it = m_lPhotos.begin(); it != m_lPhotos.end(); it++)
{
fpTotalWidth += (*it)->GetWidth() + INTERNAL_MARGIN;
}
fpTotalWidth = max(0, fpTotalWidth);
fpNewXOffset = m_fpXOffset + translationDeltaX;
FLOAT fpXLowerBound = -fpTotalWidth + m_nWidth * (1-SIDE_MARGIN_PERCENTAGE);
FLOAT fpXUpperBound = m_nWidth * SIDE_MARGIN_PERCENTAGE;
fpNewXOffset = min(fpXUpperBound, fpNewXOffset);
fpNewXOffset = max(fpXLowerBound, fpNewXOffset);
translationDeltaX = fpNewXOffset - m_fpXOffset;
m_fpXOffset += translationDeltaX;
for (it = m_lPhotos.begin(); it != m_lPhotos.end(); it++)
{
(*it)->Translate(translationDeltaX, 0);
}
return hr;
}
The following code shows how the PhotoStrip uses manipulations to move photos horizontally along the control in the ManipulationDelta
event handler.
m_fpXOffset += translationDeltaX;
for (it = m_lPhotos.begin(); it != m_lPhotos.end(); it++)
{
(*it)->Translate(translationDeltaX, 0);
}
The following code shows how photos in the PhotoStrip control handle manipulations and are constrained to vertical manipulations.
HRESULT STDMETHODCALLTYPE ConstrainedPhoto::ManipulationDelta(
FLOAT ,
FLOAT ,
FLOAT ,
FLOAT translationDeltaY,
FLOAT ,
FLOAT ,
FLOAT ,
FLOAT ,
FLOAT ,
FLOAT ,
FLOAT ,
FLOAT )
{
HRESULT hr = S_OK;
Translate(0.0f, translationDeltaY);
return hr;
}
Note how the boundary checks from the photo object will notify the application that the photo has reached the boundary. A fake touch up message is sent up and capture from the PhotoStrip is released.
The following code shows how photos release capture.
case PS_PHOTO_BOUNDARY:
if (pStrip != NULL)
{
Photo *pPhoto = (Photo*)lParam;
TOUCHINPUT tUp = pPhoto->GetLastTouchInput();
pStrip->m_pDispatch->ReleaseCapture(tUp.dwID);
tUp.dwFlags &= ~TOUCHEVENTF_DOWN;
tUp.dwFlags &= ~TOUCHEVENTF_MOVE;
tUp.dwFlags |= TOUCHEVENTF_UP;
pPhoto->ProcessTouchInput(&tUp);
SendMessage(GetParent(hWnd), PS_PHOTOSTRIP_BOUNDARY, (WPARAM)pStrip, lParam);
}
break;
The message that is generated to the parent window contains a pointer to the photo object which can then be used to render the photo to the gallery control.
The Collage Control
When a photo is moved to the collage window, the collage uses the message to create an image that it displays. The following code shows how this is done.
case PS_PHOTOSTRIP_BOUNDARY:
pPhoto = (Photo*)lParam;
pStrip = (Photostrip*)wParam;
tInput = pPhoto->GetLastTouchInput();
g_pOverlay->ReleaseCapturedCursor(tInput.dwID);
tUp = tInput;
tUp.dwFlags &= ~TOUCHEVENTF_DOWN;
tUp.dwFlags &= ~TOUCHEVENTF_MOVE;
tUp.dwFlags |= TOUCHEVENTF_UP;
pStrip->ProcessTouchInput(&tUp);
GetWindowRect(pPhoto->GetHWnd(), &stripWnd);
GetWindowRect(g_pGallery->GetHWnd(), &galleryWnd);
tInput.x += stripWnd.left;
tInput.y += stripWnd.top;
fpPhotoHeight = (FLOAT)(stripWnd.bottom - stripWnd.top)
* (1 - INTERNAL_Y_MARGIN_PERCENTAGE)
* PHOTO_SIZE_MULTIPLIER;
fpPhotoYPos = (FLOAT)(tInput.y - galleryWnd.top);
fpPhotoYPos -= ((stripWnd.bottom - stripWnd.top) * INTERNAL_Y_MARGIN_PERCENTAGE);
g_pGallery->LoadPhoto(pPhoto->GetPhotoURI(),
(FLOAT)tInput.x, fpPhotoYPos, fpPhotoHeight);
g_pOverlay->SetCursorCapture(g_pGallery, tInput);
break;
When a photo reaches a boundary in the collage window, the collage will then remove the photo from the list of objects that it is tracking and will release capture of touch input. The following code shows how this is done in the case of photos hitting a boundary.
case PS_PHOTO_BOUNDARY_INERTIA:
case PS_PHOTO_BOUNDARY:
if (pCollage != NULL)
{
pPhoto = (Photo*)lParam;
if (pPhoto != NULL)
{
pCollage->m_lPhotosToDelete.remove(pPhoto);
pCollage->m_lPhotosToDelete.push_front(pPhoto);
}
}
break;
The following code shows how this is done in the case of a boundary being passed during inertia.
case WM_TIMER:
if (pCollage != NULL)
{
for (it = pCollage->m_lPhotos.begin(); it != pCollage->m_lPhotos.end(); it++)
{
(*it)->ProcessTimer();
}
for (it = pCollage->m_lPhotosToDelete.begin();
it != pCollage->m_lPhotosToDelete.end(); it++)
{
if ((*it)->IsPastBounds())
{
DWORD dwID = 0;
while(pCollage->m_pDispatch->GetCursor(*it, &dwID))
{
pCollage->m_pDispatch->ReleaseCapture(dwID);
}
pCollage->m_lPhotos.remove((Photo*)(*it));
pCollage->m_pDispatch->UnregisterTouchTarget((*it));
(*it)->CleanUp();
}
}
pCollage->m_lPhotosToDelete.clear();
pCollage->Render();
}
break;
Conclusion
Creating a control framework for complex Windows Touch applications is the best way to deliver an experience with reusable components. A newer version of this document will be released online at http://code.msdn.microsoft.com/WinTouchPhotostrip later this month (February) so keep an eye on the @WinDevs twitter account if you are looking forward to the update. Apologies for no colorization in the C++ code, I’m working on potential solutions that will work with this blog platform.