(untagged)

Krita* Gemini* - Twice as Nice on a 2-in-1

Tim_Duncan

25 Mar 2014

This article will discuss how the Krita developers added 2 in 1 mode-awareness - including implementation of both automatic and user-selected mode switching and some of the areas developers should consider when creating applications for the 2 in 1 experience to their applications.

This article is in the Product Showcase section for our sponsors at CodeProject. These articles are intended to provide you with information on products and services that we consider useful and of value to developers.

Download PDF

Why 2-in-1

A 2 in 1 computer (2 in 1) is a laptop computer that can be transformed into a tablet by manipulating the body of computer. Laptop mode (also referred to as desktop mode) allows a keyboard and mouse to be used as the primary input devices. Tablet mode relies on a touch screen, thus requiring finger or stylus interaction. A 2 in 1, such as the Intel^® Ultrabook™, offers precision and control with multiple input options that allow you to type when you need to work and touch when you want to play.

Developers must consider multiple scenarios in modifying their applications to take advantage of and conform to this new type of transformable computer. Some applications may keep their existing menus and appearance nearly identical in both modes. While others, like Krita Gemini for Windows* 8 [11], may carefully select what is highlighted and make it available differently in each user interface mode. Krita is a program for sketching and painting that offers an end-to-end solution for creating digital painting files from scratch [12]. This article explains how the Krita developers added 2 in 1 mode awareness—including implementation of both automatic and user-selected mode switching—and some of the areas developers should be considering when creating applications for the 2 in 1 user experience.

Introduction

Over the years, computers have used a variety of input methods from punch cards to command lines to point-and-click with a mouse or other devices. With the adoption of touch screens, we can now point-and-click with a mouse, stylus, or fingers. Most of us are not yet ready to perform every interaction with touch and, with mode-aware applications like Krita Gemini, we don’t have to. The new 2 in 1s, like an Intel^® Ultrabook™, can deliver the user interface mode that offers the best UI experience possible, on one device.

There are multiple ways that a 2 in 1 computer can transform between laptop and tablet modes (Figure 1 and Figure 2). There are many more examples of 2 in 1 computers on the Intel website [2]. The Ultrabook computer can transform into tablet-mode from laptop-mode by detaching the screen from the keyboard or using another means (such as folding the screen on top of the keyboard) to disable the keyboard and make the screen the primary input device (such as folding the screen on top of the keyboard). Computer manufacturers are now starting to provide this hardware transition information to the operating system. For example, the Windows* 8 API event WM_SETTINGCHANGE and the “ConvertibleSlateMode” text parameter signal the automatic laptop-to-tablet and tablet-to-laptop mode changes. It is also helpful for developers to include a manual mode change button for users’ convenience as well.

Just as there are multiple ways that the 2 in 1 can transform between laptop and tablet modes, software can be designed in different ways to respond to the transformation. In some cases it may be desirable to keep the UI as close to the laptop mode as possible, while in other cases you may want to make more significant changes to the UI. For example, Intel helped KO GmBH combine the functionality of their Krita Touch application with their popular Krita open source painting program (laptop application) in the new Krita Gemini application. The Krita project is an active development community, welcoming new ideas and maintaining quality support. The team added the mechanisms required to provide seamless transition from the laptop “mouse and keyboard” mode to the touch interface for tablet mode. See Krita Gemini’s user interface (UI) transformations in action in the short video in Figure 3.

Figure 3: Video - Krita Gemini UI Change – click icon to run

Intel has worked with many vendors to help them add 2 in 1 awareness to their applications. The Intel Developer Zone provides significant resources to help developers build applications, including 2 in 1, touch sensors, graphics, and more. Please refer to the Related Links at the end of this article.

Create in Tablet Mode, Refine in Laptop Mode

The Gemini team set out to maximize the user experience in the two modes of operation. In Figure 4 and Figure 5 you can see that the UI changes from one mode to the other are many and dramatic. This allows the user to move seamlessly from drawing “in the field” in tablet mode to touch-up and finer detail work when in laptop mode.

Figure 4: Krita Gemini tablet user interface

Figure 5: Krita Gemini laptop user interface

There are three main steps to making an application transformable between the two modes of operation.

Step 1: The application must be ”touch aware.” We were fortunate that the touch aware movement was started well in advance of the current 2 in 1 activity, as that would have been a much more challenging transition that the simple to-and-from tablet mode. Intel has published articles on adding touch input to a Windows 8 application [9].

Step 2: Add 2 in 1 awareness. The first part of the video (Figure 3) demonstrates the automatic sensor activated mode change—a rotation in this case (Figure 6)—followed by the user-initiated transition via a button in the application (Figure 7).

Figure 6: Sensor-state activated 2 in 1 mode transition

Figure 7: Switch to Sketch transition button – user initiated action for laptop to tablet mode

Support for automatic transitions requires the sensor state to be defined, monitored, and appropriate actions to be taken once the state is known. In addition, a user initiated mode transition [Figure 3] should be included as a courtesy to the user should they wish to be in the tablet mode when the code defaults to laptop mode. You can reference the Intel article “How to Write a 2 in 1 Aware Application” for an example approach to adding the sensor-based transition [8]. Krita’s code for managing the transitions from one mode to the other can be found in their source code by searching for “SlateMode” [13]. Krita is released under a GNU Public License. Please refer to source code repository for the latest information [14].

#ifdef Q_OS_WIN
    #include <shellapi.h>
    #define SM_CONVERTIBLESLATEMODE 0x2003
    #define SM_SYSTEMDOCKED 0x2004
    #endif

Not all touch-enabled computers offer the automatic transition, so we suggest you do as the Krita Gemini team did and include a button in your application to allow the user to manually initiate the transition from one mode to the other. Gemini’s button is shown in Figure 7. The button-initiated UI transition performs the same functions as the mechanical-sensor-initiated transition. The screen information and default input device will change from tablet mode touch input with large icons to laptop mode with smaller icons and input from keyboard and mouse. However, since the sensor path is not there, the button method must perform the screen, icon, and default input device changes without the sensor-state information. Therefore, developers should provide a path for the user to change from one mode to the other with either touch or mouse regardless of the state of the button-initiated UI state in case the user chooses an inappropriate mode.

The button definition - Kaction() - as well as its states and actions are shown in the code below (Reference 6):

// Snip from Gemini - Define 2-in1 Mode Transition Button:

         toDesktop = new KAction(q);
         toDesktop->setEnabled(false);
         toDesktop->setText(tr("Switch to Desktop"));
SIGNAL(triggered(Qt::MouseButtons,Qt::KeyboardModifiers)), q, SLOT(switchDesktopForced()));
         connect(toDesktop,
SIGNAL(triggered(Qt::MouseButtons,Qt::KeyboardModifiers)), q, SLOT(switchToDesktop()));
sketchView->engine()->rootContext()->setContextProperty("switchToDesktop
sketchView->Action", toDesktop);

Engineers then took on the task of handling the events triggered by the button. Checking the last known state of the system first, since the code cannot assume it is on a 2 in 1 system, and then changing the mode [13]:

// Snip from Gemini - Perform 2-in1 Mode Transition via Button:

#ifdef Q_OS_WIN
bool MainWindow::winEvent( MSG * message, long * result ) {
     if (message && message->message == WM_SETTINGCHANGE && message->lParam)
     {
         if (wcscmp(TEXT("ConvertibleSlateMode"), (TCHAR *) message->lParam) == 0)
             d->notifySlateModeChange();
         else if (wcscmp(TEXT("SystemDockMode"), (TCHAR *) 
message->lParam) == 0)
             d->notifyDockingModeChange();
         *result = 0;
         return true;
     }
     return false;
}
#endif

void MainWindow::Private::notifySlateModeChange()
{
#ifdef Q_OS_WIN
     bool bSlateMode = (GetSystemMetrics(SM_CONVERTIBLESLATEMODE) == 0);

     if (slateMode != bSlateMode)
     {
         slateMode = bSlateMode;
         emit q->slateModeChanged();
         if (forceSketch || (slateMode && !forceDesktop))
         {
             if (!toSketch || (toSketch && toSketch->isEnabled()))
                 q->switchToSketch();
         }
         else
         {
                 q->switchToDesktop();
         }
         //qDebug() << "Slate mode is now" << slateMode;
     }
#endif
}

void MainWindow::Private::notifyDockingModeChange()
{
#ifdef Q_OS_WIN
     bool bDocked = (GetSystemMetrics(SM_SYSTEMDOCKED) != 0);

     if (docked != bDocked)
     {
         docked = bDocked;
         //qDebug() << "Docking mode is now" << docked;
     }
#endif
}

Step 3: Fix issues discovered during testing. While using the palette is fairly easy in either touch or mouse mode, the workspace itself needs to hold focus and zoom consistent with the user’s expectations. Therefore, making everything bigger was not an option. Controls have been enlarged for touch interaction in tablet mode, but the screen image itself needed to be managed at a different level to maintain an expected user experience. Notice in the video (Figure 3) that the image in the edit pane stays the same size on the screen from one mode to the other. This was an area that required creative solutions from the developers to reserve screen real estate to hold the image consistent. Another issue was that an initial effort had both UIs running concurrently, which adversely affected performance due to both UIs sharing the same graphics resources. Adjustments were made in both UIs to keep the allotted resource requirements as distinct as possible and prioritize the active UI wherever possible.

Summary

As you can see, adding 2 in 1 mode awareness to your application is a fairly straightforward process as long as you consider how your users will interact with your application when in one interactive mode versus the other. Read the Intel article “Write Transformational Applications for 2 in 1 Devices Based on Ultrabook™ Designs“ for more information on creating an application with a transforming user interface [10]. For Krita Gemini, the decision was made to make creating drawings and art simple while in tablet mode and add the finishing touches to those creations while in the laptop mode. As you are developing your 2 in 1 modes, consider what you wish to highlight in your application when presenting it to users in tablet mode versus laptop mode.

Additional Intel Resources

Notices

INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.

UNLESS OTHERWISE AGREED IN WRITING BY INTEL, THE INTEL PRODUCTS ARE NOT DESIGNED NOR INTENDED FOR ANY APPLICATION IN WHICH THE FAILURE OF THE INTEL PRODUCT COULD CREATE A SITUATION WHERE PERSONAL INJURY OR DEATH MAY OCCUR.

Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined." Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information.

The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.

Copies of documents which have an order number and are referenced in this document or other Intel literature may be obtained by calling 1-800-548-4725 or going to http://www.intel.com/design/literature.htm

Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark* and MobileMark*, are measured using specific computer systems, components, software, operations, and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products.

Any software source code reprinted in this document is furnished under a software license and may only be used or copied in accordance with the terms of that license.

Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.

*Other names and brands may be claimed as the property of others.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here